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THE MINNEAPOLIS REHABILITATION CENTER (MRC) STUDIED 
MINNESOTA MULTIPHASIC PERSONALITY INVENTORY (MMPI) SCORES TO 
SEE IF THEY RELIABLY PREDICTED EMPLOYMENT OUTCOME AND 
TREATMENT EFFECTS. IT WAS DEEMED IMPORTANT FOR AGENCIES TO 
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H. SUMMARY 



Purpose; 

Q?he purpose of this study yas to find out whether the Minnesota Multiphasic 
Personality Inventory (MMPI) could he used to measure the treatment effects 
of a vocational rehabilitation center program, and whether variables from the 
MMPI and other services mi^t be used to predict employment outcome. 

Procedures; 

An earlier study done at the Minneapolis Rehabilitation Center (MRC) showed 
that changes did occur on MMPI testings. However, because the measure of 
change was not adequate, and because there was no control group, it was not 
possible to state that these changes did occur because of the MRC services. 
To further investigate the problem and do other studies as well, a study of 
the topic was initiated, which was financed by a grant from the Vocational 
Rehabilitation Administration. Three separate populations were formed; 

Pilot; This group of 101 DVR referred clients was used in the 
earlier study. A pre- service and post- service MMPI had already 
been given. Further work with this group was to involve a third 
MMPI testing to see if the changes first seen actually held up, 
the collection of outcome employment data, and the use of demo- 
graphic and MMPI data to predict en^loyment outcome. 

Control; This group consisted of 29 clients who were referred 
to the MRC and were given two MMPI’s without any MRC services 
between testings. The purpose was to compare MMPI results T-zith 
the sample yhlch received MRC services to see if MMPI changes 
occurred spontaneously^ 

Cross-Validation: This group of 40 clients was used tc- see if 

results in the other samples held up under cross-validation. 

The first problem encountered was that of obtaining a third MMPI on the 
Pilot group. After a number of attempts at bringing the group back to the 
MRC for a third testing failed, this portion of the study was dropped. 
Securing outcome measures of employment was also difficult. Many clients 
would not respond, and those who did respond did not report complete infor- 
mation. Finally, an 80 percent sample was obtained - but the questionnaire 
used had to be shortened. This meant that, instead of having elaborate 
enplpyment criteria measuring such things as the quality and the quantity of 
employment, we ended up with a simple measure of whether or not the client 
was working at the time we got the information. 

Attempting to measure MMPI changes was the fundamental problem, and since 
no one acceptable method is currently in use, three different procedures 
were used. One consisted of a Sum of Differences score. To get this fig- 
ure, we subtracted the second testing on each MMPI scale from the first, 
and added up the figures for every scale, for a given client. The second 
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system, the Sum-»of~Squared«“Differences , involved the same procedure except 
that the differences were squared. The third procedure involved a Con- 
figural Scoring Method. This system used the notion of “critical changes”, 
which were defined as those scores which crossed the boundary of a T-score 
of 70 on the MlilPI between the first and second testing. 

A final problem was the choice of a statistic to relate the employment out- 
come data to a set of predictors which included not only the MMPI but demo- 
graphic and medical data as well. The statistic selected was the lambda 
model, originated by Guttman.> This statistic can be used with several pre- 
dictors and takes into account the base-rate* or “predicting the mean,” 
phenomenon. 

Results! 

Several MMPI scales were found to be useful in predicting an employment out- 
come, when the Sum of Differences method and the Configural Scoring Method 
were used. However, when these predictors were used in the cross-valida- 
tion sample they did not hold up. 

Further study was then carried out, using other possible predictors such as 
sex, marital status, medical status, age, etc. Again, although several 
variables were able to predict working at follow-up, the use of these pre- 
dictors in the cross-validation group did not hold up. 

An earlier study had been carried out at MRC in which the goal was to deter- 
mine whether or not changes did occur in the MMPI as a result of services. 
Significant changes in ceitain scale scores had been taken as evidence for 
such an hypothesis. However, in a replication study, (not reported here) the 
changes occurred on an entirely different set of MMPI scales. Our reluctant 
conclusions were that we could not demonstrate that the fi/lRC services would 
reliably affect MMPI scores — and the results of this study provide addi- 
tional confirmatory evidence for this conclusion. 

In summary, we concluded that the results, although promising in the pilot 
sample, were negative when subjected to cross-validation. Our employment 
outcome, as used in this unique sample, cannot be reliably predicted by MMPI 
test results, changes in MMPI scores, and a variety of demographic and other 
variables. 

Comments! 

A variety of arguments could be raised by statisticians and experimental de- 
sign specialists concerning the basic design, sampling procedures, and defi- 
nition of the measures used. This writer's opinion takes a different direc- 
tion. My own opinion is that this study failed to produce reliable results 
because none of the predictors are, in fact, directly related to going back 
to_jw 2 rk. As most Vocational Counselors can testify, people do not fail to 
get jobs because employers can spot a deviant MMPI during the interview. On 
the other hand, they usually won't hire someone who whistles in an interview. 
The MMPI measure of this annoying trait does not exist. Nor will people get 
jobs if they do not look for work. Again, although high scores on the MMPI 
depression key might make one suspicious, some pretty unhappy people do 



manage to make that first contact before 10 in the morning. It is also 
obvious to those who do keep jobs that you can believe in the second coming 
of Christ, hate your mother, and still hold a job. Provided, of course, 
you donH criticize your boss — a trait not measured by the MMPI, but still 
a c 2 ?itical necessity for those who want to keep working. 

These comments reflect a basic problem in successful prediction studies in 
the field of vocational rehabilitation. We do not have an organized system 
of measuring those beuaviors which are Important in getting and keeping a 
job . Instead, we use a psychopathological or medical model, with a few 
demographic variables thrown in for ease of measurement; then, with a weak 
set of outcome measures, we hope for good results. A few studies in the 
field (none, however, which were discussed in this study) have shown such 
good results. This study shows how such good results mi^t be obtained - 
just don’t do cross-validation. 
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IH» BACICGROUKD, PURPOSES, AND PROCEDURES OP IHE PROJECT 



A. IHTOODUCTION 

Mmeapoli^ Rehabilitation Center is a private, non-profit, conrorebensive 
vocatioml rehabilitation facility-. Its purpose i^to assess 
the vocational, social, and emotional problems of the multipj'y disabled and 
disadvantaged, Throu^ the use of a simulated work setting (workshon) and 

efforts Of an ^teraiscipUnary team 

clients* en®loyaMllty, doU readiness, and ufrk ' 
skills, tependang upon feasibility, clients are prepared for the labor 

services, or they are prepared for and 
resources. It is intended that clients ulll be better 

th^ the^tevff^S^r"" “ "" li^® 

65 > . come. from Minnesota and from nei^bor- 
socioreconanic class, and manifest many ^s 
bf P^o''=ls?> 3 ‘ ®bey are seen at a $lme vhen they '^eS to 

empl<5ment or tralhlng, but are VocaUonally 
?“® ^ combination of physical, emotional, intel- 

lect^!, and euucational problems. They are referred by many asenCies in- 

the^AlD^toSS”^ ^ vocational. Behabilitation of the StX of MinnIsL, 
ua^mp^f ^ ^termination Unit of the State of Minnesota, the ite- ’ 

partment of Employment Security, and others. 



B, HISTORY^ Airo PURPOSE OF OEE PROJECT 

Ihe Ml^esota Multlphasic Personality Inventory (llMEt) is a -widelv used 

originally developedl to discriminate 

d6™.ioLpS'*J+*^^° ®®=>' ana from normals.. Since its 

development-, it has come to be used much more broadly. It is used to evalu- 

f r functioning in the individual oase^ts uSl as^f diLrS^ 

^uns "®^ ®® vith, psychiatric ‘ 

£=i+h’ *^4^* settings other than hospitals, and mental 

vppp+Lr^^^ff,’ example, it is ejctensively used in educational and 
raS^seSinr**^”^^ The Minneapolis Hehabllitation Center (MRC) is one 

^ ■^^® “linical psychologist, in con- 
r^p+p tests, interviews, and social history,, to 

to^pL®^ functioning to a client's adjustment in the MBC program, 

ana^?ppff °“® W>rk ’Jehavlor, to suggest; treatment apprtachS, 
and to assist in vocational planning. MRC clients usually take^e *PI 

\elsh, G.S., and Dahlstrom, W.G., eds., Basic ReadlnKS on the MMPT in 
lacchology and Medicine. Minneapolis, uFiTeraity bf Minnesota^esd° 1956. 
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early in the first week of their program. (The average length of time they 
are in the program on a full-time basis is five weeks. TOien they con^lete 
this part of the program, they begin Job- seeking or training, wait for the 
beginning of the training course they have select d, return hom^, are re-' 
ferred to a sheltered workshop, etc. Early in 19 u 2 , it bec^e the practice 
to administer both ^entry" and *^e:idLt” MMPIs to clients in order to re-assess 
personality fiinctloning after MC sei^ces. 

In the ^st , the has been used at MRC as a clinical instiument. It was 
being assumed that, in our client population, personality components were 
being measured by the MMPI and , further , that personality changes would also 
be reflected on the MMPI (over two or i^fe administrations of the test). 

Such changes are also believed to be vpeatiomlly ^gnif leant. 

Given the assumptions about the relationship between treatment^ personality 
change, and vocational dutcomS— and the parallel assumptions that pSfsonality 
states, and changes therein, can be measured by the MMPI— the MRC staff de- 
cided to compare entry Mfl^I results„ with results from e^t MMPIs. In a 
pilot study carried’ out at MRC ih I963, 'entry and exit MMPI scale scores (K- 
corrected 0 >-sCaie) were compared for ior Clients (the pilot san^le). The , 
subjects were selected on the basis of whether they 

(a) had ; both an entry and exit- MtCI profile-'^ . ' 

(b) entered tie progra]n betiween Jantory, 1962 and April, 19631 

(c) conoid read at about the' fifth-grade le^l or bettefl and 

(d) had been in the program at least four weeks. 

This group, which amounted to about 25 percent of all BW referr admitted 
to MRC program services, during the time period defined, cannot be considered 
in any sense "representative” Of the ihole client grbUp seen during the 
period (tie educational requirement s,, and the requirements for being able to 
take a valid MMPI insured that the stiidy ^oups would be , to soke unknown 
degree, "above average" for the total MC population. ) Hbwevef, the purpose 
of the study precluded any possibility of such representativeness, since 
only a subset of vocational rehabilitation populations are sifted and can 
take a valid IMI. ’ ' ^ ■ " 

The pilot study established that 'iiMPI scores are sensitive to pre-post pro^ 
gram changes (see reference l). The next question was what relationship did 
such score change have to actual changes in personality dynamics or behavioral 
characteristics/ bf the c 3 -ients over the peribd of vocational rehabilitation? 

A way of .getting an. answer to this ^estioh Caii be outlined here. 

In the ori^hal application, two things were' promised — a'-keasure of psycho- 
logical change under rehabilitation and a conclusion as to the effectiveness 
of vocational rehabilitation in causing psychological change. . ^ . 

' ■' ■■ //’;.'7 J,.'" * * 

We cannot, howeyer, measure changes in. someone until, we have a measure of 

change that we trust (in the psychologist's lan^ge, a valid and reliable 
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measure of change):* .For exaK^le.,’ imagine the .situation that }j;e, had no con- 
cept of measured length, and soiiie.one came to y.c?u and said — , ?ypur child is 
two inched taller than last year.” Unless you, had. alicady accepted the 
measure dimension implied "^y "two inches taller” (i^e. , there is: a useful , 
convention called "taller than” and there is an agreed-upon metric whic^, 
has a stable relationship to the area of reality that you apply it to — 
that is/ that the' relationship between your, yardstick. and. tlie rectangular 
opening thfou^ which you drive your car .every morning -and ni^t doesa^t 
change), you would question the odd jargon which hs was . But you. 

don’t question it because you have depended upon the measure and, your car’s 
fenders are relatively unscathed. ■ : , 

Suppose then that we had a measure of "psychological state” like this, with 
a measure Of psycliologiCal change Constructed from two readings on this 
state at different times i Suppose that it "behaves well” with .respect to 
the way other constructs behave, and we trust it .in the same way we trust 
in our measurements of change in length. Suppose, for example, that we 
know SOmedne whose reading on this measure of psychologic state is "medium” 
or ”hi^" after vocational rehabilitation, and the following relationships 
hold true: he has gone back, to work. and he was out of work when the measure 

of his psychoiogical state was "low; ” his relationships .with his wife and 
children are reported to be "just, fine now'^”:and they were not good when, 
his reading was low; and his relationships with his foreman are such that, 
it is' fepOited "the.'foreman is a bear;, but. the. client seems to he able t,p- 
get the work done so that the foreman is happy,” and, preylGUsly, with the 
low reading, he was^ reported to he a hothead who had struck superiors with , 
no provocation — and so. oh, then we begin, to trust the measiure of, psycho- 
logical state-> and the., .corresponding measure, of change. 

Once we had that measure , and trusted it in the same way we trust measure- 
ments of len^h, we could then perform an, experiment to test jiov well ^C, , . 
is doing. We would take measurements at the beginning pn clients . entering ! 
the iprogram, take measurCmehts on the same vClients at the end of the pror . 
gram' — then announce., "this ciient.ds being discharged as:^^ fmuch, imprpyed’., 
"but some other client had ho improv^ent; indeed, his psychological .state 
measurem^ , is now such that he should he referred to a mental health, ce^“ 
ter for outpatient psychiatric care.” At the same time, since we know that 
people change scMewhat in their measufements. on this dimension without any 
special Ihteiventioh, w3 also take a control group, out pf e.xactly, tl^e PPP‘4~ .. 
latibn we are dealing with in the program;^ do, not intervene , with them, thpn* 
Ccanpafe the changes in each group* If the ^changes, in the, experimental , . , 
group are such that ire can Say that the difference between, their improye- 
ment and^ the control group’s impfPvenient Is^ greater thani could ha ej^ected. 
by <^ance> we accept this as some plausible evidence: that .the, MEtd treatment,, 
cauSes; impfPvemSnt. If we can depend uponi the measure .in a precise way, - 
then we Pan -evSn give the amount by which the clients were improved, by the 
ti’eatment at MRC* ' ' ' f. ■: ; , . • ; , ' / 

However/ we 'don’t yet have 'that ifieasure of .change. So,, before we. ican.as- 
ceftain how well the measura say a :MRC is doing>..in some sciantificaliy?-,. .. 
acceptable evidential fashion, we must, first define, the ^measure, relate it 
to one or mors ’'Ultimate briteria!’ in a variety of 'ways, and. show it. to. be,, i 
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usefuj Only tHen can we XLse ±t as the hasis for a. de.cislcm on what MRC is 
doing, in tems' of that meashfei. We can’t do hoth tasks at once, and. we 
must do the one 'hefbfe the other; That is, in the .psychologist .'s language, 
we must show that a measure .is valid and reliable before we can usu, it to 
measure something* else, ^ , 

As yet,, no one' has defined a good measure, of psychological change which. Is 
based upon the NMPIi Therefore,, before we cou?..d design and, carry out a 
study which would a’llo\? us to mke some decision about Ihe effects of MIC 
treatment iipoh the p^ohologlcal state of the client, we had first to de- 
velop an adequate measure of that state. 

Therefore, 'in earning out this .study, we, "chose" to do. what ve ha^. to do 
anyway i We defined a number of measures. Of psycholo,gical. staie , based upon 
the Itei, defined measures of changO in the state,, and rehited these changes 
to a particular "ultimate" criterion.’ ; . . ^ 

If the measure had been shown to be related to the criterion. defined here, 
according to the criteria of "adequate prediction" we jet out in the study, 
we could have then tried it again against several other criteria, If.it had 
continued to "work>" in a precise and accurate fashion, we would have then , 
declared that we had a measure of‘ psychological change. 

At that point > and only then , would we have been in a. position to set up an 
experiment which would have allo^/ed us. to measure the effects of ^l^tC inter-, 
vention upon the psychological state of our eliOnts* That., is, we would have 
had^a paychologicai measure of state for which we. . know what a . reading, "meant" 
(i,e,, ws would know when a reading was "good" or "bad,." greater .or lesser^ 
or whatever). 

While the approach diOcussed'aboye provided, a , rationale ..for; solving, the 
problem Of deriving a measure' Of change from! its relationship to. a criterion, 
certain other design problems remained, Ghe. particularly difficult problem 
was that of differehtieting between measures .Of change for which we can disr 
criminate effects of rehabilitation from "random changes", and those , fp^ whi,ch 
we cannot' -do thisv ■ * • ' .:r . ; , > , 

This is the ciaS^slcal es^erimental problem of ;^fferentiating, "spontaneous" 
changes from those caused by some, agent or technique of .interes.t,, ,in a 
laboratory 'environment , -^th b. prior hypothesis to be tested,, such, a problem 
could be solved by use of a sliple before-and-after design.. That is, define, 
a population of "subjects needing, intensive .yocational rehabilitation ser?*. 
vices," Select a sample of; such; subjects fr.om: the pppulatlon, Allocate them 
to an "es^rimental" group \diich is to receive treatment,, and. a ^ control group 
from which treatment is- to be withheld. Take thC; releyant measiu?ements- from 
each grOup. Then, using a. pfe-selecte'.dl criterion, decide whether the trea^ 
ment group has improved more than the control group, j j .■ . - • 

However, there are important, differences which .arite when human. groups .form,, 
the popuiatiOh^ as they do here. Chief among thes.e is. pur., abiii^ “ate. 

tain experimental contrpl^ ’Thatvis,*, ..spe.cific tp tre.aimieats auch .as. ypca^;. 
tiohal ‘'rehabilitation (;VR)^5-which ;is administered in. an^ uncontrplled enr* . 
viromnent— is that "treatment" might not really be withheld from a "control" 






group. For exwirpJLe;, suppose that a .group vere selected for an ‘^experiment," 
in *which the whole group was measured . on the IMEC at the beginning and at 
the end of a five-week, period, then measured epme tte later for their out- 
come.. One-half of the group would he ;giyen vocational rehabilitation treat- 
ment during the five-week period and one-half would not he .given such treat- 
ment. A number of questions would arise— including some about how we would 
choose to guarantee that the control group did. todeed not receive VR treat- 
ment elseTdiere during the five-week period, or following the five-week 
period but prior to the time of outccpie measurement Further, if we at-? 
tempted to make spb.cial social or economic arrangements for the control 
group, in order to' guarantee "no "VR treatment, elsewhere.," ,h,ow are we to 
answer impertinent questions, about the possible "treatment effects" which 
are induced by such arrangements.? 



Along with the question of whether we can ;witMiold VR treatment is that of 
whether we should . That is, it may be argued that, if we already have an 
efficacious "treatment," it is unethical to withhold it from those who may 
benefit i Therefore, the concept of a, control group which receives no treat- 
ment or a placebo is ethicaliy obnoxious... iprobiem has been discussed 

in the literature of drug trials,, with the following kind of rebuttal: To 

insist upon a control group .{i.e. , a basis for deciding whether treatm.ent 
group differences are fealiy attributable to the treatment) makes things 
momentarily .difflcHilt for a fflna.il group; however, to insist on, giving the . 
treatment to everyone precludes the possibility of obtaining coj^iima 
or disconfirmation of the treatment**? .efficacy,, and makes, it, possib^^^^ that 
a nonefficacious treatment will be administered to large numbers of people 
while, at the same time, process in research may be stopped (because the 
"problem "has been solved" )'.*"'Q3qlus, tK* 'whole “concept of “cHhicalTtf^ 
of selecting the lesser of two unpleasant courses. The caalogue of this 
argument coiiLd be given for vocational rehabilitation and, as the possible 
efficacy of vocational rehabilitation ‘treatments" becomes more of an issue, 
such arguments inay.be expectedV Inthi'^ study, w ; 

taking advantage of the fact that--because ‘of ah intense shortage of voca- 
tional rehabilitation professionals in the State of Miimssota--the interval 
between IVR, referral of .clients to mSc >nd the point at whi^ they could be 
accepted for sej^ces had become about equB.l to the length of time taken for 
MRC treatments (five weeks for most clients). 

The control group was therefore selected from clients in the MRC waiting 
line, since an IWI could be administered at referral from DVR, then five 
weeks later at acceptance at MRC. Such a group was mnifestly a DVR-MRC 
population, avoided the ethical problems involved in withholding treatment 
( since they re reived treatment after the second adMhist.fatio^ of the MMPI ) , 
and miniraized the kinds of alternative services which mi^t be given between 
MMPI administrations. 

Such a solution had a compensating disadvantage, however, since it effec- 
tively foreclosed anj' chance of comparing control group outcome measures 
with those taken on the experimsntal group. Therefore, •Qie function. of the 
control group was a :.:educed one— that of furnishing a check on the ineitsufe 
of psychological change defined in the follo^ng way; If the scores on the 
^ measure defined differed "sufficiently" between the experimental group and 



•Che control group,, then they were potentially useful, and should he subjected 
to further tests. If they did not diffei, they were to he ruled out of fur- 
ther corsideration, on the ground that a defined measure should satisfy the 
minimal requirement that it diffCx-s sufficiently when tahen on a trea-fcment 
group and on a non-trea-*-ment group drawn from the "same population.” 

* K 

There was yet anotheT* difficulty. We did hot come to this study with well- 
defined, already-hypothesized relationships. Much pf t^s study was a hypo- 
thesis-generating or hypothesis- seeking effort, rather than a hypothesis- 
testing one. The number of 'varlahles and ‘variahle comhinations searched in 
order to find relationships of interest was far greater than the number of 
subjects a^vallahle* This means that, even had the oiiginal set of subjects 
been obtained on a basis which satisfied the requirements of relevant 
statistical models, and had a "satisfactory" control group existed, there 
would still have been no basis for probabilistically-based inference. 

This in5>lied a further con^romise— the definition of a cross-'vali.dation or 
replication group. Then, for eveiy' measure which ^ssed the tests of the 
comparison of control group with experimental group and for which relation- 
ships of Interest could be observed in the experimental group, those rela- 
tionships could be tested within the cross-validation group. 

In summaiy, we can tabulate the kinds of groups to be defined, the purpose 
for *vihich each was defined, the data to be collected for each, and the v inds 
of statistical operations to be perfbimeds 



Group 


Purpose 


of 

, Data. - 
Coilected 


Operations to 
be Perfdmed 


Experimental 

(Pilot) 


Search for Relation- 
ships Between MMPI 
and Outcome 


Demographic 

Medical 

-MMPI 

Outcome 


(1) Define ^ 4 PX-JRelated 
Measures of Change - 

(2) Define Measure of 
Outccme 

(3) Find Statistical 
Relationships, be- . 
tween Measures of 
Change and Measures 
of Outcome . 


Control 


Find MMPI-Related 
Measures of Change 
^ich Differ In 
Beha'vior in Treated 
and Untreated Groups 


Demographic 

Medical 

MMPI 


Ifeke Comparisons Be- 
tween MMPI Measures of 
Change for Experiinen- 
tals and Controls for 
Discriminati-ve Power 


Replication 


Decide Whether 
Relationships Found 
in Experimental 
Group are "Chance" 
or "Real" Ones 


Demographic 
Medical. 
MMPI • 
Outcome 


(1) Derive Decision Rules 
For Prediction From 
Statistical Relation- 
ships Found With Ex- 
perimental Group 
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(2) Test Predictive Power 
of Decision Rule, 

When Applied to 
Replication Group 



C. PRpCEIWRES 

The original pilot experimental group was selected, according to the defini- 
tions given in the previous section* As noted before, the 101 files select-? 
ed for this group amounted to about one- fourth of those clients served dur- 
ing the sixteen-month period of Jan. I 962 to April I 963 . 

As the project got under way,, a cross-validation group was cohstmcted by 
taking all clients served by MRC between April, I963 and Jahuaiy , I965 > ahd 
finding, the subset of these which answered to the specifications under 
which the pilot ^oup was constructed. ®iere were 100 files of Sifted 
clients for which the criteria for inclusion In the study were met. ^ Forty 
of these were selected randcanly> with stratification on sex,; the result be- 
ing two groups for whom identical data were to be collected. The* size of 
these^ groups, by sex,, was as follows;" . ; 7 . ' 

fable 1 

Number of Subjects in Pilot and Cfbss-yaltdation Groups by Sex 



Group 


Males 


Feiaales ; 


;Total 


Pilot (Experimental) 


57- 




- loi 


Cross-Validation (Replication) 


22 


18 


ifO"* 


Total 


79 


62 


ikl 



Py definition, data were available on two occasions of MMPI testing. Other 
file data taken included age at time of entry into MRC, sex, IQ, marital 
status, education,, time in program, and medical'status at entry into MRC. 
These data are tabulated for ail groups ih the- appendices. 

Plans had boen made to secure a third MMPI score, so that the stability of 
change scores could be assessed. That is, for those measures of -change 
which turned out to- be "adequate under the tests discussed, mentioned in 
Section IV.B and discussed more fully in. Section the question of the 

stability of such changes in the post- rehabilitation en'\^roment_m^s con- 
sidered to be of interest. Given that rehabilitation causes s.qme, change in 
the personality structure.,, the question arises.;whether such dianges are 
enduring or transitory. A third MMPI score might have furnished some evi- 
dence which would speak to this question. 
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Further, ill order to he able to reconstruct the post-rehabilitation history 
of the client and to construct a measure of outcome, a 48- item question- 
naire wan developed, regarding job outcomes, economic outcomes, job satisfac- 
tion, social satisfaction, changes in demographic status, and the like (see 
Appendix V)*> - 



In order to collect these data, we intended to bring the pilot group back to 
MRC for a day of tests and interviews. This plan turned out to be quite un- 
realistic. ^ the time the study got underway, some of the pilot subjects 
had been out. of MRC more than three years (the median time since leaving MRC 
was 3i”3^ months), and were not interested in returning to the Center to 
spend several hours in testing and completing questionnaires without, some 
form of' condensation for the use of their time. (Three mailings to the 101 
clients resulted in only nine clients returning to MRC for tests.) This 
feeling was even stronger among, the fraction (about one-half) of the pilot 
group which was living ohtsidO of the Twin Cities. 

Therefore, the plan to secure a third MMPl score was dropped completely > and 
a mail survey technique was attempted. It (juickly became clear that even a 
48-item:.queBtionnaire was too much for this group. Therefore, the question- 
naire was shortened to what were considered’ the 12 crucial items (see Appen- 
dix v) . ~ An initial mailing of the shortened, questionnaire resulted in a 40 
percent response rate. Subsequent mailings, at four-week intervals, to non- 
respondents brou^t total response, by all methods, up to about 60 percent. 

By this time, it aeemed evident that a. case-by-case strate^ was needed. 
Other agencies, relatives, friends, or any others who mi^t have knowledge 
of the client and who might be of help in securing a response to the ques- 
tionnaire, were contacted. Personal telephohe ''calls were mde to all sub- 
jects who could be reached by telephone. After reaching a dead end with 
respect to leads, after being turned down by the client or a relative, or 
after three contacts without success, the follow-up was discontinued. Ulti- 
mately an bverall response rate of 80 percent was reached, as indicated in 
Table 2, below. : . ‘ ... 

Table 2 

Response Rate to Outcome Questionnaire, by Group ‘ 







Experimental Groups. 








Pilot Group 
■ N- ■ i 


Cross-yaiidatipn.. 

Group 

N $ 


Total * . 

N . . 


Respondents 


84"' 


83.2 


29 


- T2,5’ 




80.1 


Non-Re'spondents 


it:\ 


16.8 


ii; 


2T.5 


28- 


19.9 


Total 


ibi- 


ibo.o 


4o 


• 100.0 


I4l 


loo.o^ 







»» *»■ ■ y 1 4 Cp <a .» « ,*v ^^»V»g^rwirtk ’’mrw ~ m mt ir 



Of those classed as nonrespondents ^ it sas learned on contact with the sub- 
ject or a relative that six were employed (two of whom were in the Amed 
Forces), 5 were unemployed (including two who had been, rehbspitalized and 
one who was in a rest home), and one had died. Of the remining l 6 , eleven 
had con^letely dropped from si^t— all letters sent to all addresses tried 
were returned— and five siD:5)ly did not respond— -their letters were not re- 
turned, but they could not be reached by telephone. 

Among those classed as respondents, response was not ccmiplete for every 
question on the outcome questionnaire » The ^^.licatipns of such nonre- 
sponse are discussed more fully in the section of pfe^ctbf-criteribn re- 
lationships in Section C, of the ^Technical liepoft. The Response fates for 
individual questions, ranged fyom 50,4 percent of the responders ( 40.4 per- 
cent of all those in both ex^rimental groups) on the question about the 
amount of hourly pay currently being received, tb 97*3 percent of all re- 
sponders (78^0 percent of the total san^le) who replied tb the questions 
"Are you now working?" Results for the entire questionnaire are tab’iiated 
below: 

^ ■ ' 

Table 3 

Number and Percent of Respbndents to Individual Items: in 
Foliow-up, by Sample Group 



Pilot Cross-Validation 

GV.o^P .. . Group 

Question. — 

(Total N = 101) (Total N = 46 ) ' 





N 


Percent 


N 


Percent 


Are you Now Working? 


82 


81.2 


28 


70.0 


Number of Hours 
'Worked' per Week ■ ' 


46 ' • 


- 45 v 5 


19 


-- 47*5 


Hourly Pay 


4 p 


39.6 


17 

t 


; 42.5 


No . Jobs ' Held Since 
Leaving MRC 


74 


73.3 


28 


7 Qt 0 


No. of Months Worked 


48 


hi . 3 


21 


5.2.5 


No. Moves Since 
Leaving MRC 


80 


79.2 


27 


' , ' \ 67 ;r 5 


Change in Marital 
Status Since 
Leaving MRC 


79 


78.2 


28 


70.0 


Time "Laid Up" 
Since Lea'vlng MRC 


76 


75.2 


28 


70.0 


Change in Overall 
Living Status Sj.nce 
Leaving MRC 


80 


79-2 


26 


65.0 



I 






The problems vhich led to the partiquler kind of control ^oup used in this 
project were discussed earlier, in essence,, the plsn xo secure members of 
such a group was as follows; Counselors of the Minnesota State Division of 
Vocational Rehabilitatibn. (EVR) were to administer an MMPI to cHents who 
were prospective referrals to ivjRC, with a goal of T5 such referrals. Given 
the expected attrition between referral and entry, a loss of 10-15 subjects 
could be es^ected- Given the XfVR referral rate to MRC, it could be expected 
that such a control group would be developed within four to five months. 

Three months and three e^oirfeato:^ letters later, MRC had on file.. 11 MMEEs 
from. pVR referrals, althou^ the refeiral rate to MRC had not fallen below 
the expected rate. Of the 11, six had actually taken an entry MMH, thus 
qualifying, as control group piembers. Personal contacts with DVR counselors 
revealed that it was Impossible in many cases, and difficult in most, to do 
the neceasary client testing. Many pf the counselors work in small offices 
with lindtad testing facilities, ^ny clients never come into the office* 
contacts by the counselor are on field trips, frequently to the clients* 
homes. In such cases the counselor would be reluctant to administer the 
MMPI and wait the hour or more necessary for the client to finish it. Thus, 
for a number of practical reasons, it was unrealistic to expect individual 
DVR counselors to be able to deliver MMPI pre-tests- at the expected rate. 
Therefore , the number of cases in. the control group was smaller than ex- 
pected ( 29 } • Details on the demographic, psychological, and medical make-up 
of this group, which were collected at entry, as. ^dth the other two groups, 
are givep .in Appe.ndix I., 

As a result of^ this work, the number In each of the three groups, for each 
of the two stages of intormatipn-collecting, was the following: 

- . . . . , .. . Table 4 

Size of Each Group in the Study— by Group .and Stage 



. 1 



Stage of , . 

Information Collection Pilot Cross-Validation Control 



At Entry Into MRC 




■ ” 




(Medical, MMPI and 
Demographic Data) ' 101 


4o 




29 


After Rehabilitation 
Treatment at MRC 
(Follow-up Data) 82 


28 
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IV. IEGPIC5AL EEPORT 



A. nJTROnJCTION . ■ 

One of the original aims of this pro^ject "was to find an adequate measure of 
the client change under treatment, using the LSMPIi Much of the problem 
of candidates for vocational rel^hilitation is psychological, and a ma^or 
component of intensive rehabilitation consists of seryices; aimed at the 
solution of problems in this area. If the services cause changes in. emo- 
tional dynamics during the rehabilitation process, the may provide some 
measure pf these changes... 

If svich a measure is to lie provided^ two problems ,must be solved: 

l) Some stable, MMiPI-reiated measure of, psychological change must be 
found. 

Xl) , rlhe_ measure must be. eutahlished as "relev^t*^ or yaiid 1:^ its re- 
.3ationsh;J.p to some xa^terion. , 

B. DEmiNG A MEASURE OF : - ' 



The solution to the first problem can be , undertaken in :vjhat is essentially 
an arbitrary way . At this time there in no univeraUy accepted me.asiire of 
MMPI-related p^chological change. Iherefore we are free to choose from 
any possible l®^-based measures that, may seem useful. This entails choos- 
ing from an infinite number of possible, sporing sys 

Since we pan only deal with a, liml.t.ed number of such possible measures of 
change,,- we must choose those whic^ seem a priori tp have a hi^ prpbabiliiy 
of success in providing; a stable, adec^uate measure of p^cholp^cal change. 
Xf we do not find a satisfactory measiae of change, this doeniipt demons 
strate that one does not exist— we mky not have looked far ehou^. 

l) A single sum-of-scale-diffShences type of change measure. 



*Por exanpie, if we choose from a very restricted set of possibilities, de- 
fined as the "difference" between two administrations of a test x^th n 
questions, assuming only one system of weighting combinations of the items 
into scales and a single method of differencing between the two adminis- 
trations, there will be 2a possible score types. With the MMPI, these 
items are commonly lunped, with some questions showing up in more than one 
of the scales, into 13 scales (l4 for this project). Therefore, if we 
take only T-scale. scores for the two administrations of the MMPI, we have 
2^^ « 16,384 possible ways of getting a difference score. 
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2} A siim-of*8quares-of-6cale*differences type' of change measure* 
3 ) A configural-scoring method. 

These change measures are defined in the following way: 

1) "%e Sum of I)iffef.ehces'Me^ihbd. 









XT / , 



Here, we take the ^-scal^ kdore oh the second adnidhistra bf **6he‘*' 
test, then subtract it from the score .oh ithe first adMnlstrat^^^^ 
given" subject, these scores ahh sukaed pVer all 14' schles l\sed”'ih the project 
(ES, H, F,‘ k, *1^0; the sigh df the dLf*fereh^ ES)’, and the' ' 

sum of these l4 differences is taken as the measure of change ’ i.e. ,-”if yi is 
the T-score on the second administration of the MMEE,.for the 1th scale ^ aiid _ 
xi is the T- score bn the first adik^histratibh,’' ^ the' ditterehce score for ’ 
the Jth suhject.is. defined as ,.14. , .. . / 

' • ^ '■ - -UT’.*.: 

i«l ^ ^ 

point of view, we woulcj, “want thope who had benefited from treatment to have 
a sc6fe> "dj-'/' which' and positive*, anS those who. had no t^ 

(or those who had had treatment, but wefS" hot libli^d)' to ha^e’ s^^^ which 
were near zero, or were negative. 



2) The d , or Sum»of-Squared-Idfferencesv'Methbd l 






' generalized bistah^^'fififetibh, which hah beer pht foiterd- at 
various times as a way of applying the'* least-squaresj criterion to discrittii^ 
nant pfoblems, has soke Inthitive appeal as measure of 
profile scG^’ei ^ch;d statistic Is'' a measufs “of the "volatility” of the 
change in scores over tVo‘ administrations' in cbmpdrisoh to the' stim-Of- ' 
differences method. This becomes clear when we look at what happens to the 
changes 'Ohveach^ of tw^^ laV/ge ahd' pbsit?£v^ the other 

is large and hd^tiye, 'the hW bf^’the ' tw hear zero for the Ihm-of- 

differehces method, 'hut very lar^' fo'f ■ the" simi-bf^'kquared-diffefence¥ method.; 
This metho'd ie'^defihed iO e.xfy-iv.i - r'-t 'i" 'll - 

a= - (x - y J*" 

« «• #1 ^ P 

3) The Configural Scoring Method . 

. Ab a last way of measuring differences between two adm^uistrations - 
of the MMJl, W have'.i^es cbhfl^ral SCOre^ 



which*' seWe as .diSchiminatOfS am various kinds of personality t^es. 




constructing a measure of change '-t^S to Use the“^ notion of the critical bbi^d- 
ary which arises in mary disCdsBiohs of scofitfg:* ' Often,* Cliclcai in- 
terpreters focus their interest on whether a specific T- scale score is yo or 
over (i.e., beyond the two-signa point on ti.v3 standardized scales), or not. 



On th 6 'basis of this notion^ ” critical changes" were defined as those 
scores which crossed the T »70 'boundary 'between the first and second admin- 
istrations. Those v^ich went from under 70 to 70 or over 'between first and 
second administrations would be coded as one kind of extreme score, and 
those which went from fO or over to under 70 would be considered the oppo- 
site, kind of eybreme score. Scores which did not cross the boundary between 
administrations would be inte^reted as non-extreme. The scores were then 
coded in the following way, one for each of the l 4 scales i 

Configorai Coding 



* 


Second Administration 




Under 70 


70 or Over 


First Administration 
Under 70 


2 


0 .. 


First Administration 
70 or Over. 


" . 3. 


1 



The scores can be given ordinal meaning, with an“o" signifying, the "worst" 
kind of change, and a "3" the "best" kind of change. Unlike the first two 
kinds of scoring j^stems, in which one summed "global" score was used, the 
measure of change in this scoring system. is a set of scores; for example, a 
subject *s change measure might be the following; 

ES L F K 1> 2 3 h 5 .6 7 8 9 0 
( 0, Oi 1 , 3 , 2, 3;> C, 0 , 2 , 1 , 1 , 1 , 2 , 3 ) 

Sudb a set of scores would be interpreted as. follows; 

1 ) Kie subject had a score of less than T*-=70 on the first adi^nistra- 

tion, and a score of 70 dr more on the second, on scales ES, L, 3^ and 4. . 

2) The subject had a scoire of 70 of more, on both aMnistrat.ions, 
scales P, 6 , 7 ^ and 8 . 

3) The subject had a score of less than 70, on 'both administratlojis, 
on scales 1 , 5 > and 9 . 

4) Tbe subject had a score of 70 dr more on the first administration, 
and a score of less than 70 on the second administration, on scales K, 2 , 
and 0 . 



0. THE RELATIONSHIP' OF THE CHANGE .MEASUBE TO .A CRITERION 



The second problem, establishing the relationship as relevant or valid by 
relating it to sane criterion, can be solved "by two kinds of operations in 
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the case of global scores, and one kind of operation in the case of vectors 
of scores, -within the constraints of the s^pie size available for ’this 
study* . ' ‘ 

l) Checking for "Spontaneous ;(jhaiiKe “' ‘ 

QJhe firat kind of operation 'is -used as a simple check- criterion. In 
the procedures section, it was no-ted that a. coMon ex^rience in the analysis 
of MMPI test- retest da-fca was that of finding a change th taking the test 
(other than the common factor that the group was taking the MMPI for a second 
•time). Profiles generated from the second administration of the test common- 
ly seemed to indicate an "in^roved” profile. Therefore, we mi^t require 
that any MMPI-lDased measure 'of change during rehabilitation services should 
be sensitive enough so that spontaneous change attributable to a second ad-, 
ministration of the test' would -not be large enou^ to cloak or wash out the 
measure of real psychological change.. » 

If we suppose , as a preliminary assumption, thafe :the rehabilitatiqn process 
does indeed "cause” a psychological change, -we may then, req-uire that the 
value of Whatever measure we have defined have some discernible difference 
(when it is estimated for a "treated" or experimental ^pup) frcm the value 
it takes when another group, equiyalen-fc every respec-fc dthef than that of 
having the "treatment", is subjected to the same measurement (thS "control 
group"). If there were no difference between experimentals and controls on 
the same measure, then the measure wopld be, disqualified for,, fnrthpr. t^stin^ 
for its ^^sible utility - on the ground that , wha-beyer other ^fxsps' tha . 

measure M^t have, it' does nbt have the critipal property of,', bping a^ 
discriminate treated ,.om -untreated. r-, . , . . . 

Accordingly, a "control group" was defined, which was generated by the same" 
referral process aa_the "e:^rimentai^ ^oup" .. O^e time interval between 
testing was approximately equal (an avefagp time ' pf ' Tiy^^^ for both 

groups. The control group differed, however, in having received no services 
between the time of the two testsi, while the experimental grpup recpiye 
such, services. The administrative constraints on the design did not allow 
for a faadpmization pirocess in allocating tO: the. twQ,,^oups,. so that prob- 
lems of generalizabiiity exist. For exejsmple,. .the .ad-vaptages , pf recpiurse 
a probabilistic model are lacking. Therefore, whether the experimental 
groups and the dontf oi groups are "equivalent" with respect t.p the charac- 
teristics which mi^t be important in their differential response tp treat- 
ment is not decidable in any useful sense. Nevertheless, comparison on* 
characteristics considered routinely important .in , vocational, .rehabilitation 
is of interest. Some of these comparisons are discussed as background to 
the comparison of the groups on the MMPI-based measures of change which we 
have definedi ’Also, gfOiip mearis and standard , deviations fpr each of the lit 
scales used lii this study for each se:c^. for each, of .the two opcasipns pf ^ 
testing, and for each of the three groups studied, are given. These compar- 
isons are included as Appendix I. 

The basic method for .comparing the two groups iS: the, following , Sinc.e; we^ do 
not know how "good" any score is, we will compare the distribution of scores. 
That is , we -will look at the proportion of thoser iwithin .a ;grpup. haying, a 



score less than some value k> oyer the entire range oi k, i.ev, the cumu- 
latiye distribution function of .the group. We will then compare the dis- 
trihutipn functions of the three groups, using a nonparametric procedure 
for comparing such functions, the Kolmogorov- Mrnov (K-S) statistic. Oliis 
statistic (the greatest vertical discrepancy . between pairs of empirical 
distribution functions) is based on prob^^bility considerations, namely, 
that we assume a null hypothe^>is. which postulates a commop distribution 
•.function for the two groups, under -the randbm sampling model. If the two 
sample groups differ so much that, we would believe, on long- run’ sampling 
considerations, that this would ,be a rare e'vent given 'that the two groups 
are "really" samples from a common distribution, we then decide that the 
two groups probably come out of "differing" improvement score distributions. 
It should be noted that these groups were r.ot derived in such a way that a 
probability model can be easily assumed; nevertheless, we, can use it in an 
"as. if", spirit-, as. a basis for future work. 

Several different versions of the, sum and sum-of-squares measures were 
used, and. the analyses were done, separately for each sex, as well as .for 
both sexes at once, on the. chance tha*!! reactions on ■the measure might be a 
•function of sex as well as •breatment/non-trea'tment. Such an analysis for 
the configural scoring method was J-.ipossible, since experimental and con- 
trol groups would be needed which would be some multiple of (4)^^ in size, 
in order to test whether the two groups differed in distribution or not. 

The results , using the K-S procedure to test for differences, between ex- 
perimantal and control groups, were essentially negative. .Tests were per- 
formed for the sum of difference scores and for the sum of -squared differ- 
ence scores.. It was our intention to test also for possible sex inter- 
actions; however, the number of females in the control group was far too 
small to allow ary kind of useful comparison wheii a fair^ insensitive non 
parametric proceduc'e was being, used. Figures 1 aud 2, which are graphs of 
•fche' two measures, make clear the fact that •the empirical distribution func- 
•tions of the; three groups differ very little, and could have easily come 
from the cumulative distribution . functions of a single underlying popu- 
lation. 



Tables of the distribution function of each of these difference scores are 
included in Appendix II. Although -the distribution function of the two 
major groiips does not differ much for the sum of difference scores. Table 
1, Distribution of Sum-of -Differences Scores, exhibits s.qme interesting 
differences between groups :when •we ' .subdivide the experimental (pilot plus 
replication) gfoup according to a rough measure of relative work success 
arfter rehabilitation. We -will discuss the implications of these differ- 
ences in the next sectio>n. 

2) Finding Predictorr;Oriterion Relationships 

For those measures of change which passed the first test, a 
second test was applied.. Was the measure of change related to some "ulti- 
mate criterion" or a .cpnyenienib surrogate therefor? 

. , K * , » ' 

a) Earlier, , it was npte.d that there is at this time no widely- 
accepted measure Of MMPI-based or MMPI-related psychologi- 









cal change* One Important reason for such a rtate of affairs 
is the fart that there is no "ultimate criterion" of i» 5 )f ove- 
ment due to psychological change that is vldely accepted* 

That is, \fG heed some kind of construct to serve as a criterion 
for the measures of p^chological change vhich will tell us 
when these measures 6f change hehave In siicb a way that we can 
decide that the cshahge in the undefiying thing measured (eome 
aspect of the client* s personality) is "good" of "preferred” - 
as against when the change should he taken as an indicator of 
something "had" of "not preferred” or "no change." 

Otherwise we might make all the measurements we please, using 
any kinds of "measuring sticks” we cafe to define; hut Such 
measurements would not, in themselves, give us any information 
about the meaning of the changes in the measurements. They 
would make Sense Only when they (and 'the construct of -^iOh 
they are a part) were related in sdme "lawful," regular way to 
some other set or sets of iiieasufements (and the construct Or 
constructs of which these bthef measurements are a part) which 
alfea^ have meaning f of us. : • 

If we find a strong relationship hCtween the measVif e of chunge 
and the measurements on the criteirion, we csin then attribute, 
in a, "hackwafd inference," 1 jcci the "preferred" -states Ofi the 
efiterion hack to theif related states on the predictor.- For- 
example , if we find that virtualHy all of those whose score; cn 
some measufe of impf ovemeht- is equal to of ^eater than -150 
were also In the preferred state on the efftefioh (for e-jcample, 
went hack to work), while virtually all of those with a BcOfe 
of less than I 50 oh a predictof were in the hoh-preferred state, 
(didn’t go hack to work), we would in^ut'e tO the charactertstic 
of "having a score equal to, or gfeatef than, 150"‘ the asso- 
ciated characteristic of "improvement under treatment." 

iC . 

ito psychology, however, there are some special complications. 
First of ell, aside from the hOundh ;g5,ven by ^oss pefsOhality 
disorganization or sociopathic characteristic disorders > what 
constitutes a. "good" hehavioi^l trait and what constitutes a 
"had" one is a matter of ho little disagreement, festlng on 
vaiuational grounds which, differ much between psychologists^ as 
Weil as between most other classes of humhs. 'Therefore, w 
constitutes a change for the "hettef" Of for the "wOfse" ih any 
absolute sense is bound to be a matter of disagreement. 

We can, howevef , hsuaily secure somewhat morO agfeemeht by 
limiting our discussions of "b.etter" or "worse" to 'shai^ly cir- 
cumscribed afeas, where more people may be able to agree triihin 
the limited context ‘ of interest, , on What "better" - and 
mi^t be. This is mOst feasily dOne within the 'areas of -Hi'e 
in which constructs which specify measures •: ? preferredner^s 
exist Which are "natural" to the afea (e.g. in cefbain areas 
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Flgiire 2; Cumulative Distribution of Sums of Squares of Difference Scores on Two Administrations 
of the MMPI, For Three Vocational Rehabilitation Groups 







I 



of economic life, where the object is to maximize sales or 
profit or "xjrestii^e of the institutions”, within certain 
hounds given hy convention, taste, or principle.) 

In the area of vocational rehabilitation, we have available a 
limited but "natural” criterion of preferredness, to which we 
can (tentatively, at least) relate any defined measure of 
psychological change,. Such a choice can be rationalized on j 
the basis that, while the "therapeutic” aspects of the work ofi 
a vocational rehabilitation center. are extremely important, 1 
they are, in the context of this study,, instrumental to an * 
underlying economic purpose. Therefore, the specific cri- 
terion of interest in this study must be in some fashion re- 
lated to the economic outcome of the vocational rehabilitation 
process. 

Such an economic outcome can be defined in a variely of ways , \ 
ranging from the simplest kind--whether the client has been 
accepted on any kind of job for some interval of time— to the 
most conplex— e.g. , a measurement of the "quality” of the job 
or its "satisfactbrlness," over some longer interval, using 
monetaiy measures of changes in the client *s income stream 
(including both earnings and transfer payments) and measures 
of' outcome for the client *s status on these measures prior to 
rendering the services. 

lit mi^t be argued that ary economically-based criterion is in- 
appropriate to the validation of a measure of psychological 
change, and that such an objection would apply a fortiori to 
such a specifically- defined economic, criterion. To this, we 
can only remark that any "ultimate" criterion which woiad oper- 
ate over the whole set of human purposes is not definable at 
the present time (and probably never will be), and that any 
useful criterion we might define is conditioned by a great 
nuinber of historical, social, and economic factors. Therefore, 
any such criterion will be useful only for certain limited - 
purposes (here> for exari5)le, for the purpose of finding a way] 
of distin^shing changes for the "better” in psychological ‘ 
state from those which are not "better"). 

It should be noted that the (limited) possibility always 
exists that some measure Tdiich we find to be related to one 
kind of a criterion might turn out, on further investigation, 
to be related to other, entirely different kinds of criteria 
in sensible ways and thus furnish us with a more or less "uni- 
versal” measure of say, psychologl.cal change. Until we reach 
such a measurement utopia, however, we shall have to be con- 
tent with the fact that our measure of change is useful, at 
most - in the present instance - only "with respect to" the 
criterion to which it has been related. 

Because of the administrative and budget problems involved in 
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collecting adequate kinds of information from the experimental 
-groups, the actual .criterion measurement finally used in this 
study was On the low end of the complexity spectrum. Original- 
ly, it was planned to secure data concerMng the amount of work 
done since leaving the rehahiUtation program, and to recon- 
st^ct total earnings, so that quantitative measures of "amount" 
and quality of work following rehabilitation would be avail- 
able as a measure of **total Inprovemeht." ■ 

However > because of the low budget for case- finding and follow- 
up>- relati\*e to the amount of time and effort needed to find 
the experimental group at follow-up, the kind of near- continuous 
follow-up; monitoring needed to construct such estimates could 
not be undert:aken. Because of problems of mental retardation, 
neui’ological problems affecting remote memory, and the sheer 
inability of many clients to reconstruct a description of what 
was sometimes a. patchwork of lobs since leaving rehabilitation 
servl^ces., estimates* of amounts of time worked since leaving the 
program, as well as the amount of money earned, were considered 
too uni’elial)le to funxish useful .data upon which estimates of 
total improvement could be based;, (specific data are included in 
Section III e C.). As. a' compromise, the simplest dependable 
measures available were taken; Had the client ever worked 
since treatment? Was he working at the time of follow-up? 

Of the two measures generated by these quest.ions, the one con- 
cerned' with work .status at time of follow-up was considered 
j- better than that associated with whether the client had ever 
worked since his receipt of rehabilitation services. Virtually 
all of those who had. received MRC searyices had worked at one 
time or another (85 percent), since- leaving MRC, thus making 
this measure less -useful, for discriminative purposes. On the 
other hand, ‘54 percent of the experimental group had been found 
working at the time of follow-up a much more useful percentage 
for discriminative purposes. However, discriminative purposes 
are not the- prime criterion here— the real question is whether 
those found; working at follow-up can be considered a separate, 
well-defined' gro^P, and whether this group can be considered 
• different from, those not worl^ng at the time of follow-up. 

Such a question can.be answered in favor of a measure of those 
working at follow-up in this way? Few o£ the clients (rou^ly 
about 10 percent) appear to have worked more than 90 percent of 
the time since they left MRC, Thus, at times, even some of the 
best graduates of. the pro, gram are currently out of work. 
Therefore, at any one point in time that a follovr-up survey is 
carried out, there is a non-zero probability that some of these 
clients -will show up* in the "currently not working" 
group. At the siune time, some of those graduates who have 
worked hardly at all, but have worked a little, may be working 
at the time of follow-up and may be misclassified as "more im- 
proved , since they show up in the group labelled "currently 
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•working How6V6r, "bhe probability «is are much hi^er that 
those who have the best wprk records idll show up in the "not 
currently^ working" group. Therefore, m this study, we have 
used the currentlv;- working" versus- "currently not working" 
dichotomy in a number of crucial areas. 

An analysis of the data having to do with the percentage of 
time worked by each client since leaving MRC confirms this 
point of view. Of the 8l persons for whom there were data in 
expef Rental group I, h 2 . were working at follow-up, 26 others 
had worked at some time since leciving WRC, but were not work- 
ing at follow-up, and l4, clients had never worked since leav- 
ing MRC,( see Table 5)« 33ata were not available on amount of 
time worked for l4; of the k 2 working at follow-up and for 7 "of 
the 26 who had worked since leaving MRC, but Were not working 
at follow-up; therefore, percentages of time worked had to be 
imputed for those for whom no data existed. ^ Depending upon 
the kind: of estimator used (median, or unwei^ted mean of per- 
centages) and the Mnd of imputation made for missing data 
(worst assumption! "those with missing, data worked zero per- 
cent of the time"; best assumption; "those with missing data 
wOxked 100 percent of the time"), those found to be working at 
follow-up had worked, "on the average," h3 to 90 percent of 
the t^e, with a "best estimate" of .55-67 .percent of the time, 
■sipce leaving MRC. Using the same approach for those who were 
hob working at fbllow-up, but had worked since leaving MRC, we 
find that these had worked, "on the average," from 22.5 to 
46.5 percent of the time, with a "best estimate" .of 27*5 to 29 
percent of the time, since leaving the rehabilitation program. 

Since eveiy estimate for the percentage of time, worked since 
leaving MRC for the "now working’* group is superior to every 
es’^imate for the "worked but not now working" group, except 
for* the case in which we make the very best of all possible 
assumptions for these not now working and, simultaneously, the 
worst of all possible assumptions for those currently working, 
we conclude that errors of misclassificatioh which m igh t arise 

llo-tvkrr If ft- tf 

— i^i-Oup ao our superior, cri- 

terion group will be moderate. 

If ~we consider only the groups for whom data on percent of 
time worked since leaving MRC are available we can make an 
estimate (perhaps heavily biased) of the size of the possible 
misclassification error due to using the "now working" cri- 
terion. This error would appear to involve no more than 25 
percent of the group for whom data are available (i.e., less 



• • concise outline of the problem of probabilities of mis- 

classification and their costs, atid some useful solutions, 
see Cronbach,'L.J., and Gleser, Goldlne G., Psychological 
Tests and Personnel Decisions . pp. I-29, 2nd~Edition, Univer- 
sity of Illinois Press, 1965.' 



than one-fou]rth of the "now working" group had worked less than 
40 percent of the time since leaving MRC, and less than one-fourth 
of the "not working^ hut have worked'* had worked kO percent of 
the time or more since leaving MRC; these groups constituted 
only one- seventh of the total "now working" group— after the 
"no percent- of- time-worked" data were included— and one seventh 
of the '-ever worked" group). 

VJhile the problems of misclassification occasioned here are not 
crucial, the problem should nevertheless be taken, into consider- 
ation in any future planning of studiles of thS.s type. A con- 
sideration of the probabilities of being found, at -Work at any 
given moment, given that one is a non-successfulror successful- 
graduate of a vocational rehabilitation "treatment" argues for 
monitoring post- rehabilitation enrployment at least twice in any 
kind of fbllow-up which is intented to yield Serious conclusions. 
In this way, the probabilities of misclassification of clients 
with respect to their post-rehabilitation success will be mini- 
mized, even :Lf information is secured only at the point of 
follow-up (ojr course, the follow-up time points need to be 
^judiciously selected). 



Table 5 

Percent, of Time Worked Since Leaving MRC for Pilot Group 
Follow-up Respondents, "By Work Status at Time of Follow-up 



<jo of Time Worked 


Now 

Wcivdng 




Worked 


Total 


No Information 


l 4 


7 


• ' 


21 


0.0 


•» 


mm 


14 


l 4 


0.1 - 9.9 


2 


5 


mi 


7 


10.0- 19.9 


2 


1. 


mm 


3 


20.0-29.9 


1 


4 


-> 


5 


30.0-39.9 


X 


5 




6 


40.0-49.9 


3 


1 


■ - 


4 


50.0-59.9 


6 . 


I 




7 


60.0-69.9 


1 


1 




2 


70.0^79.9' 


2 


1 


mm 


3 


80.0-89.9 


3 


mm 


mm 


3 


90.0-99.9 


7 . 


r . 


■ 7 , 


7 


Total Respondents 


42 


26 


ih. 


82 
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b) Choosing a Measure of Reljatlonshlp 



Tflien ve choose a measure of relationship for the conditions in*? 
volved in this problem, there are some difficulties Involved, 
ilfst of all, we are dealing (possibly) with more than one 
variable predicting to a criterion; the criterion is a non- 
quahtitative dichotomy (working at follow-up versus not working 
at follow-up); and, finally, the predictor variables are poly- 
tombus (either dichotomous or quartotomous. in the problems 
dealt with here). This implies that we will be dealing -jd-th a 
contingenbjr table in n dimensions (with ii the number of pre- 
dictor variables utilized). Prior to the mid-1950*s, no theory 
was avaiMble to allow us to deal with an n-dimensional contin- 
gency table in such a way that we could make some measure of 
the degree of association. And, until the last five years, no 
theory and associated methods were avalleible such that sampling 
problems could be dealt with (the early -^rork was concemed only 
with the case in which we are dealing wilh the whole popula- 
tion). Today, there are several methods available. Of these,, 
a useful model seems to be the lambda model, originated by 
Quttmah and developed by Goodman and Kruskal. Essentially, 
this is a predictive model, which is strongly analogous to a 
multivariate linear regression model. 

We can define the model in the following w^. If we have N 
subjects for whom we are making a. prediction of whether they 
are A or. B, we have the following kind of matrix: 



ICTCAL 



PREDICTION 





A ■ 


fc-% 




A 




^12 


“ll’*’“l2 


B 


“21 


^22 


“21**°22 




“iiSi 







Nhen we make the prediction, we will predict that some people 
are A and some are B. Some of these predictions will be cor- 
rect (n^^ + Ngg out of N of them; i.e,, were predicted to 

Ajs and were A, n22 were predicted to be B^s and were b); some 
will be incorrect (ni 2 +n 2 i out of N of them; i.e., ni2 of 
them were predicted to be B^s, but were A, and U 2 i of them were 
predicted to be A, but were B). 



Now the p roppr t ion of co.rr ect predic tions (n_- + 
wbab we will define to be dur HIT eMIj* 



is 



As we look at this measure, two things are obvious: l) 



If we 



flip a fair coin as each person comes up for prediction, ve 
■will predict about 50 percent correctly (i;e., (nii+n22)/N = 

.50) in the long run. This level of prediction we call CHANCE 
(50:50) PBEDICTION. ICherefore, any information -which would 
lead us to predict worse than ‘50 percent, correctly .in the long 
run is worse than no information at all; we would he better 
off flipping a coin. This means that any. information we use 
would be useful only if it helped us to do better than chance 
prediction. 2) There is one piece of information, which is es- 
pecially helpful in allowing us to predict correctly. Suppose 
that we Imow before predicting that 65 percent of the peo-ple 
are always A. Then we could simply predict, for every person 
for whom we must predict, that he will indeed be A. We will 
then be correct 65 percent of the time. The Wei of prediction 
attainable by Imowing this one piece of information, the level 
of average occurence, we call BASE-RATE PREDICTION. 'It is some- 
times called predicting the mean. - It should be easy to see 
that if, in actuality, A*s and B*s are divided 50:50: then 
chance prediction and base- rate prediction will givei the same 
results; but, as the actual proportion of -A*s depart.s from the 
50-percent level, the hit rate results obtainable fi-om base- 
rate prediction "become better and better. . 

Therefore, while hit-rate is a good measure of success, it is 
not entirely satisfactory, since we can do no worse than 50 
percent correct at ary time, and no worse than the base-rate, 
if the base-rate differs from 50 percent. A more satisfactory 
measure of our success in prediction would be a measure of how 
much better we do than the level achievable by chance or the 
level achievable by predicting the base rate. 

Such a measure is LAMBDA B» If we have information on which we 
are basing our prediction, we may measure the value of our pre- 
action by use of Lambda B, which is defined as: '^The decrease 

x'n the ^^probability of predicting incoirectly, as we add infor- 
mation . Then, if a new piece of information (over and above, 
say, the base-rate) does not allow us to predict more accurate- 
ly than if we ^ust knew the base rate, then it is not useful 
information. . . 

All of the measures defined here are , given their definition for 
population (as opposed to sample) statistics in an article by 
Goodman and Kfuskal in the Journal of the American Statistical 
Association, pp. 732-764, December 1954. Asymptotic distribu- 
tion theory is derived for these statistics in a 1964 article 
by the authors in the same journal. 

Under the conditions of this study, the relationship of base 
rate, hit rate, and lambda is the following; 




TeAile 6 

Relatlon^p Among Three Predictive Measures, 
MRC-MMPI Stu^, for Pilot Exjperimentai Group (K=8 s) 



Number Guessed 
Correctly 


Base . 
Rate 


Hit 

Rate 


lambda 


42 


.512 


■.512 


.000 


45 


.512 


•5^9 


.075 




.512 


.598 


.175 


53- 


.512 


,646 . 


.275 


57 


.512 


.695 


.375 


61 


.512 ' 


. .tW" 


.475 


65 


.'512 




.575 


69 


. . 5,12 


. .841 


.675 


82 


.512 


1.030 


1.000 



As is clear from Table 6, . lambda takes values from, zero to 
1.000, being equal to zero when additional information allows 
us to predict no better than the base rate., arid 1.000, when 
additional information allows us perfect prediction, moving be- 
tween zero and one at a uniform rate between the base-rate numr 
her and the total- group number. This procedure has other de- 
sirable properties, which are discussed in the basi.c article ip 
which it is defined. 

c) Statistical Results 

The check on the relationship between controls and the two 
experimental groups showed that the, supii of squared differences 
score seemed to be of little use in discrii^ating the two 
groups. The sum*? of- differences score,, while of little use for 
total groups, could be show to be discriminative in other ways. 
In this section^ we discuss the results, of relating this latter 
score and the configurai scoring methods to the job- outcome 
criteilon. 

(l) Sum of Difference Scores Method . Althciu^ total experi- 
mental and control group differences were of no interest, 
the distribution of scores by the work status of the ex- 
perimental group versus the distribution of the control 
group scores was of interest. The data in Table 7 below 
would seem to imply a relationship between score and 
quality of treatment outcome for those in the experimental 
(pilot plus replication) group. That is, the worse the 
post- rehabilitation work record (or, if no data could be 
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secured, and we assume the post- rehabilitation work record 
of the "no data" group to he much worse than the average — 
so that we assume that the no- data group is more like the 
"no work" than the "work" group), the greater the propor- 
tion of "had" (negative) difference scores (over two ad- 
ministrations of the MMPi), Further, the score distribu- 
tion of the "untreated" (control) group seems to he more 
like that of those who were "unsuccessfully treated" (those 
with had work records) than those who. were- "successfully 
treated" (those -with good work records)* 

Table 7 

Proportion of Negative Sum of Difference Scores Over Two Administrations 
of the MMPI— Experimental (Pilot plus Cross-Validation) and Control Groups 





Proportion of Group Having Negative 


Scores Sum 




Number In Group 


Number Having 
Negative Sum 
of Scores 


Propoirbion 


Es^erimental Group 








Working: at Follow-up 


- 57 


17 


.281 


Has Worked Since 
Leaving MC, But Not 
Now Working 


35 


12 


•3te 


Has Never Worked 
Since Leaving MRC 


.18 


9 


• 

o 

o 


No data 


31 


16 


•51T 


Control Group 


29 


17 


.585 



Such results as the above should he taken only as indica- 
tive, since ‘the likelihood of their being out of "truly 
different" distributions of score differences is not ex- 
ceedingly great. Nevertheless, they are at least sugges- 
tive for further 3:esearcii (which should be conducted at 
the MMPI item level, rather than the scale level, to avoid 
some of the problems associated with finding useful linear 
or nonlinear relationships when only scale scores are avail- 
able). 

The test of these scores by the predictive criterion shows 
again that scores as defined here are only of weak interest. 
Using these scores to predict post- rehabilitation work 
success for the- experimental group yielded a hit ratio of 
sli^tly better than 55 percent, or sli^tly better than 



3^te' prediction (lanibdQ ** •10.) • If the assumptions 
of ‘the fandom model vere acceptable., we. would ha^e a prob- 
abili% of getting such .a value about .25 percent of the 
time, if the sample lambda came from .a populat?,on lambda 
equal to zero (i.e., the measiire adds no information). 
Therefore, this measufe, as it is presently defined, has 
little promise. 

(2)’ Gohfigural Score Results . The use of conflgural scoring 
produced the most' iiseful MMEC-relatsd result of this study 
Using the configur«il scoring techiiique, a number of ccmi- 
Uinatiofis of four 14MPI scale Scores were found for tThlch 
the hit rate was T^)-7'5 percent (i.e.*, if the outcomes on 
each ^ttern of response were to hold good over repeated 
applications of th€5 rule, we could use the pattern out* 
comes to predict work outcome as based on idMPI measured 
improvement outcome, and be correct in our predictions 
' 70-15 percent of- the tlme)i This implies a san^le lambda 
in the .40.50 range. The furthejc. improvement in predic- 
tion resulting from usC of five-variable, and eix-var3.able 
Ccmbinations was considered so small as to not be woith 
the Increase in complexity of inteip)reting and using re- 
sults. 

Unfortunately > even fouf-varlahle ;^tte3rns are rather un- 
wieldy, when each of the vailahles is scored four ways 
(i.e., for each variable the scoring is either, ”0” first 
test less thsin second tost ^eater than or equal to 

T»70; ’*1” first and second test hoth greater tban or equal 
to T=70| ”2” hoth first and second tests less than T==70; 
and,, test .greater than or equal to T=70 and 
, second test less than T=70), with (4)^ = 256 possible 
. score pajbtems involved. 



With a sample, size no larger than. K=82 in the pil.pt group 
(the number in the pilot group for whom complete informa- 
tion. was ai^ailable)-, there will be a large number of empty 
patterns, and thus an unstable relationship. Therefore it 
would be useful to find a decision rule which will satisf^f 
three criteria; (a) It will have a hi^ lambda value; 

(b) it will be stable over a number of replications (that 
is, whatever decision rule we .derive. for predicting 
whether someone will work or not, given that he has a cer- 
tain improvement patteim, the rule must hehave in approxi- 
mately the same way each time we use -it*).; (c) the rule 

should he usahle by the worker who is responsible for in- 
terpreting the test and should, therefore, be' simple 
enough to i3S8 '3.h a variety of s.ituaticn.s. 

In cuc-h Uiieful. measuavj.e of change, 

we ’m'7c Fucovr.(i>:'.t',.w. iii der.;.v1.?,i.g rules rati-sfying the ' 

and third criteria, but not the "second. We have 



taken spme, of the most complex rules ,, vhlch give us a hit 
rate in. the ‘ 70--75 percent range / and; de^riyed simpler rules, 
■which. give us a hit rate, in. the 65-70' P6.rcent range. Un- 
fortunately, cross-validation ^of these rules ■with our rep- 
lication group results, In a hit-rate no better than chance 
.or base-rate prediction. 



One of these rules is the following: Consider four scales, 

■ t K, 1 (Hypochondriasis)-,. 7 (Psychasthenia-)}, and 9 (Hypo- 
mania) in combination, ihen (a) if ai^ of these is coded 
: ”0”. for a. client>; ^predict that the client -will not be work- 
•' ing at follow-upi (b). for the remaining subjects, predict 
that ail of those having a ”3" "coded on any scale will be 
found working at. fpllow-up; (c) of those- subjects now re- 
maining,- predict that all subjects ■with a "2" in the Hypo- 
condriasls scale, and a ”l" in at least one other scale, 
will be found working at follow-fup: (d) for the remainder 

(the majority of which will usually b^ those who, on all 
four scales-^ began under 70 and ended there ) , predict that 
they ■will not; be found working at follow-up. Using a rule 
of this simplicity, a hit rate of het-^-er than 70 percent 
can .be achieved. . ^ 

If we denote each separate prediction group as (a) - (d), 
the details of this result are as follows: 

Table 8(a) ■ . 

An MMPI Pollo'w-up - Employnient .Prediction Rule, 

Using. Scales K, 1, 7^ and 9 



Prediction Group 


Prediction 


Actual Outcome 


at Follow-up 




, , , 


Working 


Not Working 


(a) All ■viith *0* on at 
least one scale 


”Won»t'Work’* 


i ' 1 , 

’ 2--'- 


10 


(h). All remaining ■with 
a *3* on at least 
one scale 


."Will Work" ' 


t * 

■■ ■ . l6 • ■ 


3 


(c) All remaining with a 
^2* bn Hypochondriasis 
and at least one 
scale scored *1* 


"Will Work" 


6 


2 


( d) All remaining 


"Won’t Work" 


• 17 : 


26 


Total 




kl 


hi 



Such a decision rule results in two kinds of correct pre- 
dictions and two kinds of errors, as follows: 



Ta*ble 8(b) 

Results of Use of an MMPI Employment-Prediction Rule 





Predicted. 
Will Work 


Predicted 
Won’t Work 


Total 


Actually Working at Follow-up 


22 


- 19 


hi 


Actually Not Working at Follow-up 


5- 


36 


hi 


Total 


27 


55 


82 



The two Mnds of correct prediction (predicted working at 
follow-up; found worldLng at follow-up, n=22; predictf -1 not 
working at follow-up; found not working at follow-up, 
n=36^ total correct is 58 out of 82, or JO percent) result 
in much Improvement over the base-rate prediction level. 

However, when tiiis decision rule was cross- validated, only 
l4 of the 28 in the cross- validatior group were correctly 
classified,, or rou^ly about what we could expect by using 
chance or base- rate prediction (i.e., lambda = O.OO). The 
sfliall number in the cross-rvalitotion. ^oup for whom com- 
plete data were avilable (n=28) might have something to do 
with this; but the rule, if it is a good one, should oper-i- 
ate well with smaller groups; and, under a random sampling 
model, the result we obtained should happen only rarely, 
if the decision rule were a true discriminator. With such 
mixed evidence (a hi^ lambda on the original rule, but a 
zero lambda on the cross-yalidatipn) , it would seem that 
such lules should be investigated carefully for jiossible 
improvements - especially in the cases for whom the rule 
creates errors in decision, but should not be used in aiy 
practical application without much further research. A 
more detailed discussion of reasons for deterioration of 
results under cross-validation follows the next section. 

(3) Using Other Information : A natural accon^niment to 

questions of prediction thrpu^ measures of change under 
rehabilitation is the question of the efficiency of infor- 
mation about various states of the client at entry into 
rehabilitation as predictors of outcome. In a sense, 
such questions have less theoretical significance, since 
they are not attempting to find relationships in which 
psychological theory concerning the relationship between 
improvement under treatment and later outcome is the issue 
at hand. Here we are simply Interested in the relation- 
ship between the client’s state at entry and his later 
work state. With such measures, we are not adding infor- 
mation about the response to treatment, or even assuming 



that there JU one , no assun^tions are made about the 
effects of treatment. 

0?he follOTdng variables were coded: sex, entry score on 

each scale* of the MMPX (less than X=70 on a scale versus 
T=70 or greater on a scale), marital status (single, sepa- 
rated or divorced,, .versus -widowed or marritl), number of 
week's in program (less than five ^eeks versus five weeks 
or more), IQ (less than 100' versus 100 or greater), age 
(less than median age for group, versus greater than median 
age for .group; the piio-fc group me^an was 2k years, the 
replication group median 31 years, and the control group 
median 31 years)., medical. status (see Appendix V for ‘ 
material on this coding). 

Because only dla'‘''^om6us pf edictbrs to a dichotomous cri- 
-beribii were invox^red, dealing with these variables in- 
volved less complexi-fcy than the quairbotomous variable de- 
fined earlier for a measure of change. Here, even a six 
•variable combination of dichotomous predictors involved 
far fewer patterns (64) than a four- variable combination 
of quartotomous predictors (256 patterns). 

in looking for sets of best predictor combinations, we 
examined demographic and medical variables separately from 
the MPI variables, then examined all -variables together 
to find best predictor combinations. It -was notable that 
MMPI -variables, -taken alone, are lowest in predictive 
power. The common, deino^aphic variables rank hipest, 
with the psychological and medical variables ranking 
second. 



,^e bes-t and- second-best combinations of one, two,..., 
five or six variables are listed together, by hit rate of 
the best combination:- (See -following page) 

It is interesting that even one -triable gives us consider- 
able ^provemeht (-ujabda = .275) ih information for the 
predic-bion of outcome. Here, our prediction would be 
simple: if female, predict that the client -will not be 

working at follow-up;, if .male, predict that the client will 
"working at follow-up (the hit rate in5)roves from .51 to 
.65). After the first variable -the rate of increase be- 
comes almost constant for each additional variable added 
(about three percent additional predictive efficiency per 
additional variable, for the non-MMEE variables). The 
addition of the MMPI variables in combination pro-vides a 
considerable increase in predictive efficiency. For ex- 
ample, the combination of five MMPI, medical, and demograph- 
ic }^riables provide about 3.5 percent higher predictive 
efficiency than the combination of five non-MMPI -variables 
.(•.8p5;Yersus..768). The wiables taken alone were 
rather inefficien-fc; the best six- -variable- combination pro- 
vided no more predictive efficiency than the best four- 
variable medical- demographic combination. 
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Tatle 9 

Best Predictor Combination Related to the Outcome 
(Employed atTolIo^up Aifber Ileceip^ of MRC Services) 
By Himiber and ^Cype of Predictor Variables Used 



Number of Hit 

Variables Rate Best Variable Combination Second-Best Combination 



1 (without 

MMPi) , 6 k 6 Sex . ; ■ iferital Status 



2 

3 

k 

5 

5 



(without 

MMPI) ^ ,671 Sex, IQ 
(without 

MMPI) .707 Sex, Age, Marital Status 



Sex, Marital Status 

Sex, Congenitial Medical 
Disability, IQ 



(without 

MMPi) 

(without 

MMPI) 

(MMPI & 
Other) 



Sex, Motor. Disability . _ Sex,. Psychiatric Dis- 

.732 Psychiatric Idsability, Age ability. Sensory Di.- 

ability, iQ 

Sex, Motor Disability Sex, Motor Disability, 

.768 Psychiatric Disability, Psychiatric Disability 

Age, Weeks -in Program - -Age, IQ ; 

Sex, Age, Marital Status, Orthopedic Disability, 

.805 and MMPI Scales 2, and 8 Sensory Disability, 

Marital Status, and MMPI 
^ Scales ^4, and 7 



6 (without 
MMPI> 

6 (MPI 
alone) 



Sex, Congenital Disability, Rot Calculated 
.817 Motor Disability, Age, IQ 
Weeks in Program 

.732 MMPI Scales K,l,2 '3,7,9 ' .MMHC Scales SS,K,1,3,7,9 



As with the measures of psydiblogical change, a decision 
rule using these cdabihations should meet two basic re- 
quirements in order to be of further Interest: (a) it 

must be amenable to wide Use', d£d (b^ it must hold up 
under crbss- validation. ■ 

with dichotomous coding the use of one variable implies a 
prediction for each, of two groups; two variables imply 
predictions, for foiir groups^ etc. We -felt that any finer 
delineation- of such small samples wohld not yield 2 Tu1ss 
with ary probability of standing up over repeated use . 

As an exanple of how Just <bne variable may give excellent 
improvement over base rate , with the addition of another 
variable giving further ir^fovemeht (but at decreased rate 
of Improvement), consider ‘the’ hse of sex and IQ in predic- 
tion of Outcome. • • 
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If ve consider sex alone, ve derive the prediction table; 



Client !l^pe 


Working 


Not Working 


Percent 


Number 


Number 


Working 


Male 


'29 


16 


6 k 


Female 


13 


2 k 


' 35 



Using the prediction rule - if male, predict work; if fe- 
male, predict no work - we arrive at a hit rate of .65. 

if we now add another variable, IQ, we get a {sli^tly to- 
proved) hit rate of .67. 



Client Q^pe . 


Working 

Number 


Not Working 
Number 


Percent 

Working 


Hi^-IQ Females 6 




• 60 


Low-IQ -Females 


7 


20 


26 


Ni^-IQ Males 


12 


6 


67 


I«ow-IQ Males 


17 


10 


63 



The variables discussed above were not the only variable 
combinations whidb gave some indication of predictive 
power. The hi^-MMEC result, as previously noted, produced 
a hi^ hit rate (.732), with a consequent excellent lambda 
value (.45). However, .this was a six-variable result (K, 
1,2, 3,7,9). A stoplXfication of' Wto result produced a 
rule which utilized only three of the variables (K,3,7). 

The rule; For all clients having a K-scale score equal to 
or ^ej^ter than !F»70, or a K-scale score of less than 70 
and only one of the other two equal to or greater than 70^ 
predict' that they will be found working at follow-up; for 
all other clients, predict that they. will not be found 
working at follow-up. The resultant is a hit rate of .67, 
a not intolerable loss, given the sin^Mfication. 

There were also stoplifiable combinations of medical, demo- 
graphic, and MMPI variables. A five- variable result, which 
^ve. a hit rate of ,8p5 using (a) orthopedic handicap in- 
volved or not; (b) sensory handicap Involved or not; (c) 
single versus ever married; (d) under T=7^ versus ’JO or 
over on Psychopathic Deviate Scale; and (e) under T«70 
versus Ts7P or o-^r on the MMPI Psychasthenia Scale was 
collapsed into the simplest of decision, rules; If the 






subject had neither of the handicaps, ■was single, and had 
a T- score of less than 70 on "both scales, predict that 
the subject would not be found -working at follow-up; 
otherwise, predict that the subject would be working at 
follow-up. The result was a hit rate of .72, the best 
result of all "collapsed decision rules” defined for the 
study. 

The result of the prediction was the following: 



« 


Predicted 
Work at 
Follow-up 


Predicted 
Wo Work at 
Follow-up 


Total 


Actually Working at Follow-up 


30 


12 


42 


Actually Not Working at Follow-up 


11 


29 


4o 


Total 


4l 


4l 


82 



In the cross--validation of the last three simplified 
rules, the results were again no better than chance or 
base-ra-fce prediction, indicating that these rules also 
add nothing to our discriminative or predictive power. 



On the Deterioration of Predictor - Equation Results; The general deterio= 
ration of results which we have experienced in using these rules could be 
ascribed to one or a combination of reasons: for example, our cross- 

validation group mi^t not "really"' be from the same population. As we 
noted before, those chosen for each group do not satisfy any 3?andom- 
selection model. Since the pilot and replication groups were taken at two 
differing time inteirvals, it is quite possible that the population of those 
being served had chsinged in some basic fashion (it is known, for example, 
that different economic conditions, which induce differing degrees of un- 
en^loyment, resulu dn somewhat different groups appearing at MRC from the 
same agency). As an indication of this possibility, the median age shifts 
considerably, as between pilot and replication group clients who reported 
outcome data (from 2k to 31). Thus, the "young male” group of the pilot 
group, which contributed heavily to the proportion employed at follow-up 
for the pilot group (more than two- thirds of this group, Thich amounted to 

one-fourth of the pilot group, were en^^loyed at follow-up is non-existent 

in the cross-validation group. Yet, the cross--validation group achieved 
the same proportion of the total group en^loyed at follow-up as the pilot 
group (about one-half). 

Lastly, and probably most relevant to the matter, the search technique used 
to find "best predictive combinations" of -variables is a computerized form 
of a practice first noted by R. A. Fisher in his famous paragraph 2^1- of 

The Design of Experiments (pp. 57-58^ 5th Edition, 19^9), and the subject 
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of a number of well-lmown soluticns since (e.g*, Sche£Ce?s a posteriori tech- 
nique). The problem was first discussed within the context of the randomized 
blocks design, in which means for more than two treatments were being compared. 
It is wo3?th quoting: 

"Congparisons, which the es^eriment was designed to make, may, of 
course, be made without hesitation. It is con^rispns suggested 
subsequently, by a scrutiny of the results themselves, that are open 
to suspicion; for if the variants are numerous, a comparison of the 
hipest with the lowest observed value, picked out from the results, 
will often appear to be si^if leant, even from undifferentiated 
material, properly, such \ihforeseen effects should be regarded only 
as suggestions for future experimentation, in which they can be 
deliberately tested. To form a preliminaiy opinion as to the strength 
of the evidence , it is sometlmefa useful to consider how many similar 
conrparisons would have been from the start equally plausible. Thus, 
in comparing the best with the worst of ten tested varieties, we have 
chosen the pair with the largest apparent difference out of 45 pairs, 
which mi^t equally have been chosen. We mi^t, therefore, require 
the probability of the observed difference to be as small as 1 in 900 ^ 
instead of 1 in 20, before attaching statistical significance to the 
contrast.” 

Analogously, in order to find ”best predictor cembiiiations” of variables, 
we have, in the case of the IWi c^nge measures, searched 1,001 combina- 
tions; in the case of the MMPI entry scores, taken alone, we searched 3,003 
combinations; for the demo^aphic and medical variables, taken alone, we 
searched 210 ccMbinations; and, for the MMPI entry scores plus the demograph- 
ic arid medical variables, we sear^ed 42,504 combinations. In order to 
achieve An actual .Q5 significance probability, \!e would need tp utilize (in 
the absence of the Scheffe resiOLts) a nominal significance probability rou^- 
ly on the bfdar of p less than , 000001 (if we had, satisfied the probability 
assumptions ■ in the way we chose our original. group). !l&is provides the 
rationale for the cross-validation group, since it serves as the ”futiuc^ ex- 
perimentatiph” group, in which these ”unfbreseen effects.. .can be deliberate- 
ly tested.” .To the extent that we can ”trust” the cross- validation group, 
then, we can aecejpt the negative evidence supplied, by it, in trying out the 
decision mies derived from the many- combination search of the pilot group. 

Such deterioration in predictive power indicates that we have, not yet found 
the Rosetta Stone in discriminating, successful from unsuccess:^! at the 
point of selection for vocational rehabilitation or even aft.er MMPI measure- 
ments are made on the client *s change during the rehabiiitjation process. 

In the other two published ”success prediction” projects in which vocational 
rehabilitation outccMe of MMPi predictors were used, not m.uch greater suc- 
cess resulted. The more successful was that of Gough, Wenk, and Rozyn^o, in 
which a lambda of .l6 (our calculation from the author *s data) was achieved 
under cross-validation, using a combination of the California Base Expect- 
ancy Scale and the California Persons^ity Inventory (a combination of these 
two plus the was approximately as powerful; i.e.,, the IMI added nothing 
here) to predict success or failure in parole outcome. In the other pub- 
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lished study, DeMam developed an el^t-varlable equation for prediction of 
success in vocational rehabilitation outcome. In cross-va.lidation, using a 
group split 84:16 on the rehabilitate d-not rehabilitated outcome measure, 
the equation predicted only 65 percent correctly, or far less than base 
rate prediction. 



Such results as were found here and in the other studies cited would seem 
that, if useful predictive equations are to be derived, such 
that theory is also satisfied, we must turn to the operational definition 
of q i ^stantive variables, as opposed to the "available-demographic” or test 
^riables which are usually those investigated (as was the case here). 

That is, further investigation should concern itself with those vocational 
rehabilitation workers who are themselves "good predictors" of client suc- 
cess, and an attempt should be made to derive a heuristic programming of 
the decision rules that these workers are using in their operations. Such 
a simulation-model approach would seem, given its success in other areas, 
the most promising one available at the present time. 
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SIGHIFICANCE OF THE RE^TS OF OHIS RESEARCH 



Every vocational rehabilitation center staff faces the following basic 
problem: A constant stream of clients come to it, some of whom it can help 

very much, some of whom it can help somewhat, and some of whom it cannot 
help at all. 

The problejn for the center staff is to discriminate betveen these groups so 
that it can - under all conditions of demand for its services - refer else- 
where those potential clients whose treatment would be a waste of time and 
the public’s money. _ For example.,, it is not useful for the center to accept 
borderline referrals who can probably dp quite well occupationally without 
the help of the center, or those potential clients who are at a level such 
that the services of the center’s staff will not. help the client at all. 

Under meet conditions, it is also useful to be able to distinguish - among 
those accepted for services - those who will benefit less from the staff’s 
services from those who will benefit more from such services. 

There are at least two reasons for wanting to do this. One is that the 
agency, once it has performed its “differential diagnosis," may be able to 
allocate potential clients into different programs. For example, those who 
would benefit from the agency’s services, but have a higher probability of 
■job success, can go directly .into the agency’s "intensive .eeryices.” Some, 
however, of the lower probability- of- success group will need a short period 
of preparatory work before they wi^. be able to take full advantage of the 
agency’s intensive services. There mi^t also exist a group with even 
lower probabilities of success, who would need rather long-term preparatory 
services before they could enter the intensive vocational services. 

Secondly, given the particular social commitment of the agency, based upon 
Its ideas about what subgroup it can work best with, or what the relative 
benefits are to society in dealing with the hi^er and the lower probabil- 
ity-of- success groups, the agency will change its admissions policies some- 
what under different conations of demand for its (presumably limited 
amount of) services. 

For example, if the agency had decided that it could do its best ^ob with 
the hi^er and medium probability- of- return- to- work groups, it would then 
wish to screen out the lower probability group for referral elsewhere, or 
for delay in rendering vocational services until demand slackened. Or if, 
contrafiidse, the agency had decided that its best work was done with those 
having low probability of employment, and it believed that hi^ demand for 
its services was positively correlated with labor shortages, it mi^t focus 
on taking low and medium-pfobability-of-success group and filtering out the 
hi^er probability applicants for services, on the ground that, in a time 
of labor shortage, many en5>loyers are more than happy to serve as “voca- 
tional rehabilitators” for these “easier" cases. Whatever the conditions 
of demand for service or the agency’s social policy, then, it has the need 
to distinguish between types of potential clients, according to the poten- 



tial 'benefit the agency's services will have for them, ' A tabulation of what 
might he a vocational rehabilitation agency's possible actions in its 
admissions-decision policies, given these three kinds of determinants, is 
given below 

Table 11 ‘ ‘'f ■ 

The Possible Admissions Screening Actions of a Vocational Rehabilitation 
Center, Given Its Social Policy, Demand for Its Services, and the 
Probability of the Potential Client's Returning to Work 



SOCIAL policy OP AGENCY 


Take Those Most Likely 


Take Those Who Need 


to Succeed, If There 


Services Most, If There 


Is a Choice 


Is a Chbice ’ *’■ 



DMAKD CCa!JDITIOK" ‘ DEMM) C0I!?DITI0W - 



PROBABILITY ‘ 
OP criEKT's 
RETURNING 
TO WORK 


Great 

Demand 


Slack 

bemand- 


Great 
■ ' 3)emand 


■'Slack ■ 
Demand 


Work' a Ce 2 > 
tainty with- 
out Agency's 
Services 


Do Not Accept 


Do Not Accept 


3)0 Not Accept 


3^ Not Accept 


Hi^ Probabil- 
ity of Work 
with Agency 
Services 


Send Strai^t 
to Intensive 
Team Services' 


Send Straight 
To Team> 
Services’ • .. 


Do Not Accept 


Send Strai^t 
to- Intensive 
Te^ Sendees 

. i * * 


Medium Prob- 
ability of 
work with 
Agency Ser- 
vices 


Send to a Send to a- 

Short Prep- Short Prep- 

aratory Course aratory Course 
Prior to Entiy Prior to Ent:^ 
-Into Team Intq ,Team. ' 

Services. Services 


Send ip a 
Short Prep- 
aratory Course 
Prior to Entry 
3jato Team 
Services 


Send to a 
Short Prep- 
aratory Course 
Prior to Entry 
Into Team 
.Sei'vices 


Low Probabil- 
ity of Work 
with Agency 
Services 


Do Not Accept 


Send to Ex- 
haustive 
Preparatpiy 
Course 


'Send to Ex- 
haustive 
Preparatory 
Course 


Send to Ex- • 
haUstive' 
Preparatory 
■Course 


No Work a Cer- 
tainty Even 
with Agency 


3)0 Not Accept 


Do' Not Accept 


3)0 Not Accept 


Do Not Accept 



Serifices 



We have discussed the kinds of choices to he made and some determinants of 
these choices. Wow, we need some methods of making such choices. That is, 
we need to know how to recognize which potential clients should go into 
each path. If we are to allocate potential clients into each of these 
paths in some efficient way, we are really mking a prediction about 
whether they will work or not and under which conditions of rehabilitation 
they will have thei:^.,bp^st chance of returning to work. Therefore, if we have 
some method of predicting the pQtential client *s probability of returning 
to work, given the agency services, and can do this with entry data alone, 
we have a method for allocating the potential clients to their proper 
paths. The attempted development of such a method was discussed as the 
last portion of the technical report. . . 

We would also like to know how well Vjarious clients have done during reha- 
bilitation, so that we can predict, after making, seme observations or 
measurements of their progress in the agency *s programs, their probability 
of returning to work. A method for making such measurements would permit 
us to know which clients to keep on for further services (because they have 
not ^ec "responded to treatment”) and which ones to let go. In this way, 
we ckn make nibre optimal use of our facilities,, not, keeping oh those who 
show themsel\es ready to leave and keeping pn those who have shovm them- 
selves not yet ready to leave. , , ' 

It is to the study of this problem that the main .portion of this research 
was directed. Since it was believed that a major coopiponent of vocational 
disability and a major component, of positive changes reside in the person- 
ality dynamics of the client., the chief effort of .the study was devoted to 
examining possible measures of client change under treatment, based upon 
MMPI data, in order to develop such a method. It .was also reco^ized that, 
the development of a successful meas\jrement method would have important 
Implications for MMPI theory. 

The design of the study insured that the critical decision on whether we 
had found bases for predicting success in, the two (ilfferent cases - pre- 
diction of post- rehabilitation success given what we Itoow of the client at 
entry, and prediction of post-rehabilitatioh suc.ee s.s given a measure of 
change under treatment which we could observe, after some time in treatment - 
would be made on a conservative basis., &e results were mixed. 

In both cases, types of information were defined which gave considerable 
improvement iii predictive power. However,, in ho case was there, a success- 
ful cross-validation. Since, cross- validatiori results are cfuPial under the 
assumptions and procedures used in this study, the major results are nega- 
tive. It should be noted, however, that problems pf construction of each 
of the groups used here (pilot, control,, and, cross- validation groups) may,, 
be responsible for the negative res.ults. Further, as noted in the techni- 
cal discussion, there were difficulties in construction of the criterion 
of "success” which may also have contributed to this result. The major 
effect of this study, in this respect, therefore is to provide the basis 
for a well- designed study which could produce the theoretical and practical 
results, the need for which, was clarified. by this, study. • 
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Aside from the main results of the study, cer.*tain valuable secondary results 
were produced. Among these: 

1) One of the major information problems in vocational rehabilitation 
arises from the fact that the majority of applicants for vocational reha- 
bilitation are multi-problem cases. However, most information in the field 
concerns only the "primary problem” of the client. A multiple-problem cod- 
ing system, with rules for coding, was devised as pait of this study, and 
found to be relatively successful.* 

2 ) Althou^ the "base-rate prediction trap” and methods for its avoid- 
ance have been Imown to the literature of prediction in general and voca- 
tional psychological matters for some years (dating from Paul Meehl*s article 
in the mid-fifties), it seems to have been honored more in the breach, in 
research reports where it was a relevant consideration, than in its appli- 
cation (see, for example, the two reports of research on prediction discussed 
in the technical report, of this study), The problem is defined in operation- 
al terms, ”^th examples., here. A simple statistical method for solving the 
problem is. also given, using Guttmann’s lambda, which is a measure of the 
decrease in the probability of predicting incorrectly, over and above that 
given by knowledge of the base rate of occurreL'je of a phenomenon, as we add 
pieces of information. Althou^ the associated distribution theory was used 
here in only one example (because of the underlying probabilistic problems 
Involved in the way the study groups were constructed), confidence integral 
methods for this statistic, deirived by means of asymptotic .theory, and appli- 
cable on a fairly small sample basis are available for application in prob- 
lems where they are relevant. Therefore, in any study satisfying design as- 
sumptions, we may compare prediction methods for relative predictive power, 
decide whether the predictive^ower of an equation developed throu^ these 
methods is "significant^itidSfjKrent from zero”, and the like.* 

3 ) Some methods of definition of a "measure of psychological change” 
during vocational rehabilitation were explored, and a promising form of con- 
figural scoring was developed. As was noted in the technical report, the 
possible number of such measures which may develop is extremely large. 
However, the results obtained here pointed to s<mie further interesting possi- 
bilities using scale score results. Future research in this area should be 
planned to take advantage of known "common configurations" and the personal- 
ity information which is generally considered by MMPI theorists to be 
correlated ^Tith these configurations i Such research would then allow 
measures of change which have been validated by a correspondence with an 
economic criterion to be related to current psychological theory. 

4) A major problem in evaluating what goes on during the rehabiH cation 
process has been the lack of a well-defined criterion of success. Some of 
the complexities inherent in the use of a criterion which involves only the 
judgpient that someone is working at time of follow-up (but not necessarily 



*(Note: Information on these methods, and computer programs which simplify 

their use, is available from the project’s biostatistician, Mr. Copeland) 
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much "before or much afters i*e., one of the problems in "success” is 
that some of its components are stochastic) were examined, so that 
what is involved in using a criterion at the low end of the "com- 
plexity spectrum" mi^t he somewhat better understood. The problem 
is difficult enou^ so that the problem of constructing a criterion 
deserves "on-site" research, in which the post-;rehabilxtation prog- 
ress of the client is closely monitored, so that a quantitative 
criterion measure can be constructed on a large- sample basis. Only 
then can we accurately assess the predictive relationships between 
entry information, "in-process" information, and the criterion. And 
only when we have done this can we derive useful entry- predi ctors-of- 
success and useful measures of change under rehabilitation. 
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-APPENDIX I 



TABIES AND DISCUSSION OF CHARACTERISTICS OF EXPEfelMENTAL 

(pilot and repucation) and control groups 




In a number of other tables in this report, we will report data concerning 
the subjects in the bivariate or trivariate modes, as well as the more usual 
univariate mode. Because the data is presented in this fashion, far more 
Information is more available for any inspection of subgroup data that the 
reader may care to make than is possible in the usual mode of presentation 
in which all data is presented only in the univariate mode* 

1* Age, Sex, and Education of Experimental and Control Groups * It 
will be noted that the data for the 101 subjects in Experimental Group I 
(the Pilot Group) and the 4o subjects of Experimental Group II fthe Eepli- 
catlon Group) have been grouped together for these comparisons (see Table 

Although the sex ratio for the e35)erimental groups is fairly even (about 
56:44), fewer than a quarter of the control group (? of 29) were female. At 
the same time, the average age for the control group is somewhat greater 
(sli^tly more than a fifth of the experimentals are under 20 years of age, 
but only 1 of 29 controls)* ‘While the control group distribution of years 
of education was somewhat better than that of the experimental group (one- 
half of the experimentals had 12 years or better, versus about two- thirds of 
the control group with equal attainment), the sex differences within groups 
were somewhat greater* Males in the experimental groups were the worst in 
this regard. 

2* Age and I.Q * The distribution of IQ is about equivalent for con- 
trol and experimental groups (see Table 2). “Whether there is an age- IQ 
interaction in the control group is not really decidable— but th^^ existence 
of sue! an interaction for the experimental group is clear* Those accepted 
for services prior to the age of 40 are clearly a low- IQ group (about 80 
percent of those under 20, about 70 percent of those between 20 and 39, but 
only about 30 percent of those 40 and over have IQ scores of less than lOO). 
The effects of this age- IQ interaction on work after rehabilitation services 
•yill become evident later in this section. 

3« Marital Status, by Group * A last demographic comparison between 
the two groups illuminates one especially interesting characteristic of the 
groups entering MRC--the heavy preponderance of single versus ever-married 
among the clients* Here, those classified as single amounted to about two- 
thirds of the experimental group (98 of l4l) and one-half of the control 
group (l4 of 29). Only about one- sixth of the experimental group (25 of 
l4l) and one- fourth of the control group (7 of 29) were listed as married 
and currently living with their spouses* 
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Table 3 

Marital Status, By Group 



Jferital Status 


Group 




Experimerrbal 


Control 


Single 


98 


Ik 


Married 


25 


7 


Separated 


6 


2 


Divorced 


11 


4 


Widowed 


1 


2 


Total 


Ikl 


29 
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APMDIX II 



MEANS AND STANDARD DEVIATIONS OF FOURTEEN MMPI SCAI^E SCORES (T-SCORES), 
TAKEN OVER TVJO OCCASIONS OF TESTING, FOR THREE GROUPS (PILOT GROUP, 
REPLICATION GROUP, CONTROL GROUP), BY SEX 



Table 1 

Cumulative Distribution of Sum of Difference Scores (Zxi) on Two Administrations 
of the MMPI, for Three Vocational Rehabilitation Groups 
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Table 3 
Pilot Group 



Scale 




Ifele 


(H= 5 T) 






Female 


(N= 44 ) 




Initial Score 


Final 


Score 


Initial Score 


Final 


Score 




Mean 


S.D. 


Mean 


S«D. 


Mean 


S.D. 


Mean 


S.D. 


ES 


50.!^9 


11.06 


51.05 


11.47 


48 . 61 ; 


7.70 


52.30 


9.11 


L 


53.77 


8.80 


53.33 


8.60 


53.75 


8.15 


55.07 


8.93 


F 


58.65 


12.85 


57.5^ 


12.21 


56.41 


7.87 


55.68 


9.23 


K 


55.07 


10.75 


55.51 


11.13 


53.32 


8.38 


55.80 


9.67 


1 


58.42 


14.36 


56.67 


13.50 


55.36 


12.66 


52.36 


12.68 


2 


62.42 


13.53 


59.61 


13.4o 


60.89 


11.50 


58.68 


12.24 


3 


59.89 


11.95 


59.‘^9 


10.30 


56.55 


10.70 


55.14 


11.36 


hr 


64 . 4 o 


n. 4 o 


62.35 


10.77 


59 . 3 h 


11.25 


61.86 


n.29 


5 


56.44 


10.34 


55 . 0 if 


9.83 


52.14 


9.21 


50.14 


9.21 


6 


57.95 


10.94 


56.79 


10.53 


57 .i ^5 


8.78 


55.66 


10.64 


7 


60.67 


10.70 


58.60 


11.74 


58.57 


9.05 


56.57 


8.63 


8 


62.93 


13.52 


62.74 


i4.28 


60.32 


9.79 


59.11 


10.62 


9 


57.42 


12.11 


57.77 


11.67 


56.95 


10.35 


57.20 


11.99 


0 


52.84 


10.19 


52.77 


10.18 


60.50 


9.93 


57.25 


10.35 
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Table 5 
Control Group 



Scale 




Male (N=22) 






Female (D‘:= 7 ) 




Initial Group 
Mean S.D. 


Final Score 
Mean S.D- 


Initial Score 
Mean S.D. 


Final Score 
Mean S.D. 


ES 


46,77 


12.42 


48.05 


l4.80 


^^ 5 . 5 T 


11. r" 


47.00 


15.45 


L 


53.00 


8.13 


53.95 


9.03 


52.00 


10.63 


52.29 


12.93 


F 


59.82 


16.54 


62.23 


18.69 


6k . h3 


15.58 


62.29 


10.59 


K 


55.23 


10.40 


58.14 


12.24 


59.00 


11.80 


57.71 


11.86 


1 


60.68 


10.28 


64.41 


13.27 


53. T 1 


5.94 


52.86 


5.01 


2 


63.82 


12.48 


65.27 


15.65 


63. T1 


16.26 


67.57 


16.68 


3 


60.86 


9.19 


64.09 


10.38 


61.00 


6.06 


59.86 


10.09 


k - 


65.27 


11.05 


69.14 


i4.05 


69.71 


13.38 


68.00 


14.59 


5 


57.50 


11.37 


59.95 


10.96 


47.29 


7.61 


49.29 


11.70 


6 


60.68 


11.54 


62.23 


16.23 


69.57 


12.54 


67.14 


9.86 


7 


61.77 


11.80 


64.59 


15.09 


63.43 


11.12 


56.00 


17.68 


8 


65.41 


16.56 


68.77 


17.24 


63.57 


10.61 


68.43 


12.82 


9 


57.05 


l4.o8 


62.32 


10.92 


57.57 


14.35 


53.14 


13.69 


0 


52.73 


8.88 


51.00 


10.51 


55.29 


13.97 


55.43 


12.01 
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APPENDIX III 



EXEERB«EAL GROUP (PILOT A35D CROSS- VALIDATION) OUTCOME DATA, 
BY AGE, SEX, EDUCATION, IQ, MD MARITAL STATUS 




Data are included separately for the Pilot and Cross 
Validation groups, categorized by outcome (now work- 
ing versus has worked - hut is not now working versus 
never worked versus no data on follow-up), in the 
following tables; 

1) By Age and Education 

2) By Age and, IQ 

3) By Age and Sex 

4) By Iferital. Status, and Sex 

5) By Five Collapsed Variables of Interest 
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Table III- lb 

Lrosfs- Validation Group - Outcome Status 
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Cross-Validation Group - Outcome Status 
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Table III- 5 

Some Relationships of Interest Between Client 
Characteristics and Outcome 



Months out of MRC and Outcome - Pilot Groun Only 


. 


Months Out 
of MRC 


•* 


Outcome 


• 


Working at 
Follow-up 


Wot Working 
at Follow-up* 


Total 


16-28 


15 


6 , 


21 


29-38 


27 




61 


Total 


h 2 


ko 


82 



b). Marital Status and Outcome - Pilot and Crossr Validation Groups 
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i , 


Working at 
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c) Sex and Outcome Pilot and Cross-Validation Groups 



, 


. 


Outcome 


; 


Sex ^ 
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Total 
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Table III - 5 (Continued) 

Some Relationships of Interest Between Client 
Characteristics and Outcome 

d) I.Q. and Outcome - Pilot and Cross-Validation Group Ifeles Only 
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Total 


"Average" (9O-IO9) 
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Education and Outcome - Pilot and Cross-Validation Gr( 
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APPENDIX IV 

MEDICAL CODING FOR REHABILITATION CLIENTS 



o 




Finding useful methods of characterization of the medical status of clients 
in vocational rehabilitation centers has alwys been difficult— and no -well- 
accepted diagnostic coding scheme for such problems exist* Yet, the 
client *s status, vis-a-vis such problems can make the difference bet-ween 
successful and unsuccessful rehabilitation » The project staff gave parti- 
cular attention to the development of a short, easy medical coding scheme 
-which would have some predictive -validity for the kinds of populations seen 
in comprehensive rehabilitation centers. As -will become e-vident later, the 
attempt had a fair amount of success and should be useful in future research 
in this area. 

The coding sr^'-eme developed was essentially a dichotomous one, coding a yes 
("l"), no ("0 , "no information” or ”no decision possible” codings in 

answer to -fche -x llowlng questions: 

1. Is any medical disablli-fcy of the client progressive? (versus, of 
course, -whether it is stable). 

2. Is any disabili-fcy congenital? 

3* Is there a motor disability? 

4. Is there mental retardation? 

Is there an orthopedic disability? 

6. Is there psychiatric involvement? 

7* Is there a sensory Involvement? 

The rules for this coding scheme were developed by one of the authors, and 
were the following: 

!• Progressive versus Stable . Here, if the disease -was known a priori 
(e.g., Parkinson* s Disease, Multiple Sclerosis) to be progressive, it -was so 
coded; otherwise, some reference to the progressive nature of a diagnosed 
disabili-fcy had to be given. Or, to be recorded as stable, the same kind of 
indications had to be present. In general, such disabilities as residuals 
of poliomyelitis, mental retardation, cerebral palsy, and traumatic hemi- 
plegias were so categorized. In this coding, because of lack of decision 
informat?^'', or inability to make the decision on medical grounds, such im- 
pairmentf. as the following could not be classified: chronic ulcers, 

seizures, psychiatric disorders, obesity, emphysema, chronic brain syndrome 
due to alcoholism, back injuries, CNS disease, arthritis,- glomenpjone- 
phritis. 

CoPf^gtiital versus non- Congenital * Any Impaiment which -was by def- 
inition congeiil-fcal, or -was present or noticed at birth, -was classified as 
congenital. This included some convulsive disorders, congenital orthopedic 
deficiencies, dwarfism and the like, \toere symptoms of a disorder had 
appeared at a definite time, it was coded as non- congenital. When etiology 
is unkno"wn, or there is not enough information, the coding was "indetermi- 
nate" - which included, for this project, some, mental retardations, MS, 
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Friedrich’s staxia, and CNS disease. 



3. Motor versus non-Motor . Problems in movement, muscular or neuro- 
muscular weakness or poor control (e.g., back strain, spasticity, paralysis, 
and arthritis) were included as motor. 

4. Retardation versus non-Retardation ^ Any client whose highest re- 
corded IQ was less than 80 was listed as retarded. 

5. Orthopedic versus non-Orthopedic . Any involvement of bone struc- 
ture, bone growth, fractures, deformities, amputations, arthritis, dwarfism, 
back problems, and like, were included. 

6. Psychiatric versus non-Psvchiatric . Any history of mental illness 
or hospitalization for mental illness was included. 

7. Sensory versus non-Sensory . Any sensory disability (e.g., blind- 
ness, tactile disorders) v;as include.K. 

It should be noted that a number of multiple codings for one particular 
diagnosis could occur. For example, back problems and arthritis would be 
categorized as motor and orthopedic. 

As it turns out, for both experimental and control groups, the progressive- 
stable and sen sory-non- sensory codings included only a few cases (8 cases in 
the experimental group and none in the control group were listed as progres- 
sive, while ten cases in the experimental group and only one in the control 
group were coded as sensory). If we then consider only the four other dis- 
ability codings (psychiatric, retardation, motor, and orthopedic - lumping 
the "indeterminate" and "non" codings), and attach the congenital description, 
when needed, only one- seventh of the experimental s (21 of 141) were without 
some disability coding (see Table III-I). A little less than half of the 
group (67 cases) was coded as single-disability. The remainder, about three- 
eighths of the experimental group (53 cases) were coded multiple-disability 
with a slightly heavier weighting toward psychiatric cases and toward 
single-disability cases the only differences of note. 
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Table IV-1 



Distribution of Disability- CcMbinations Among Experimental 

and Control Groups 



Experimental 


Control 


No Disability Listed 


21 


h 


Psychiatric Only 


36 


1.1 


Orthopedic Only 


3 




Congenital Orthopedic Only 


2 




Orthopedic and Psychiatric 


1 




Retardation Only 


9 




Congenital Retardation Only 


3 




Retardation and Psychiatric 


1 




Congenital Retardation and Psychiatric 


1 




Motor Only 


26 


6 


Congenital Motor Only 


9 


2 


Psychiatric and Motor 


5 




Congenital Motor and Psychiatric 


1 




Motor and Orthopedic 


14 


4 


Congenital Motor and Orthopedic 


k 


1 


Psychiatric, Motor, and Orthopedic 


1 




Motor and Retardation 


1 




Congenital Motor and Retardation 


2 




Motor Retardation and Psychiatric 


1 




Motor Retardation and Oirthopedic 




1 


Total 


l4l 


29 
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Taljle IV-2 



Medical Coding Data, Bivariate Distribution of Congenital Nature 
of Handicap and All Other Handicaps, Pilot and Cross Validation Groups 









Pilot Group 


Cross-Validation 

Group 


Coded As: 




Coded as Congenital? 
Yes No Total 


Coded As Congenital? 
Yes No Total 


Progressive 
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6 
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1 


1 


in Nature? 


No 
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91 ^ 
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3i^ 


39 




Total 
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35 




Motor? 


Yes 


X5 


38 


53 


1 


10 


11 




No 
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kl 


k8 
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25 


29 




Total 


22 


79 


101 


5 


35 


4o 


Mental 


Yes 


5 


8 


13 
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Retardation? 


No 


17 


- 


88 
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31 


35 




Total 


22 


79 


101 


5 


35 


ko 


Orthopedic? 


Yes 


5 


15 


20 


1 


k 


5 




No 


17 


6k 


81 


k 


31 


35 




Total 


22 


79 


101 


5 


35 


ko 


Psychiatric? 


Yes 


2 


26 


28 


- 


19 


19 




No 


20 


53 


73 


5 


16 


21 




Total 


22 


79 


101 


5 


35 


4o 


Sensory? 


Yes 


1 


6 


7 


1 


2 


3 




No 


21 


73 


9k 


k 


33 


37 




Total 


22 


79 


101 


5 


35 


ko 
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CONFIDENTIAL 



MINNEAPOLIS REHABILITATION CENTER 
1900 Chicago Avenue 333-2335 

FOLLOW-UP QUESTIONNAIRE 

Please complete this form. Most of the questions have several choices with a short 
line after each choice. Wherever you see this, please make an X on the line after 
the answer which Is the most correct for you, Mark only one answer . There are a 
few questions which ask for an explanation In your own words but most of these 
questions can be answered simply by marking X, 

1, Are you now working? V es N o If no, why not? 



2<. Do you work? Full-time ^Part-time It varies 

3, What Is your pay? $ 

Is this per hour? 

Is this per week? 

Is this per month? 

4, How many jobs have you had since leaving MRC? 

None One Two to Four Five or more 



Is It take-home pay? 

Is It before taxes pay? 



5. 



Please add up the amount of time you worked on ail the jobs you held since you 
left MRC and write that total time here, 

months 



6, Have you moved since leaving MRC? No Yes - How many times? 

^Once ^4 times 

Twice 5 times 

3 times More than S ■ 



7, Has your marital status changed since leaving MRC? No Yes 

Got married ^Got separated 

Got divorced \Vlfe or husband 

died 

8, Have you been laid up with illness, Injury or for any other reason for more 

than two weeks since you left MRC? No ^Yes - For how long? 

2 to 4 weeks ^4 to 6 weeks 6 weeks to 2 months 

2 to 3 months 3 to 4 months 4 to 6 months 

^6 to 9 months 9 to 12 months More than 1 year 

9, Since you left MRC, have things: gotten better 

gotten worse 

stayed the same 

10, What, if anything, did you like about MRC; 



11, What, If anything, did you dislike about MRC?. 



12 



Have you any other comments? 



Name Number 



Address 

City & State 



The form on the following pages is to be used as part of a study on people 
who came to the Minneapolis Rehabilitation Center. Only the people doing the 
study will see this form, and your name will not appear on it. 

Please write your correct name and address at the top of this page. When we 
receive your completed form you will be given a number in order to identify 
your form to the people doing the study and this page will be torn off and 
kept in a locked file apart from the form itself. Your number will be known 
only to these people doing the study and v;ill therefore keep unauthorized 
people from knowing who filled out your form. 

Please answer all of the questions truthfully. None of the statements you 
make will affect your position with your family, employer or anywhere else 
that you might not want people to know what you said. 



Thank you for your cooperation. 
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MINNEAPOLIS REHABILITATION CENTER 
1900 Chicago Avenue 333-2335 

FOLLOW-UP QUESTIONNAIRE 



CONFIDENTIAL 



Number 



Please complete this form. Most of the questions have several choices v/ith a short 
line after each choice. Wherever you see this, please make an X on the line after 
the ansv/er which is the most correct for you. Mark only one answer . There are a 
few questions which ask for an explanation in your own words but most of these ques- 
tions can be answered simply by marking X. 



1. Are you now working; 



Yes 



No If no, why not? 



2. If the reason is you cannot get a job, please explain why you cannot get one. 



3. If you are working, where? (MRC will NOT contact any employers named) 



4. What kind of business, company or organization is it? 



5. What is your job title? 



6. What exactly do you do on your job?^ 



7. Do you work? full-time part-time 



10 . 



it varies 



8. If you work part-time or if your time varies, do you usually work; 

more than 40 hours per week 2 0 to 30 hours per week 

a bout 40 hours per week 10 to 20 hours per week 

^30 to 40 hours per week ^less than 10 hours per week 



9. What is your pay? $ 



Is this per hour? 

Is this per week? 
Is this per month? 



Js it take-home pay? 

Is it before taxes pay? 



If you are working, how much is your present job like the kind of w,'»rk that you 
planned for during counseling? 

Exactly what I planned for in counseling ^Somewhat the same 

Much like what I planned for in counseling ^Not at all the same 

11. If you received training, did you use your training on any past or present job? 
Very much ^Somewhat _ Very little ^No training received 
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12* What do you like most about your job 
will NOT contact any employers.) 

Like Most 

The work 

^Hours 

Pay 

^The boss 

Co-workers 

Easy to get there 

^Other (explain) 



and what do you like least? (Remember MRC 

Like Least 

Tne work 

Hours 

P ay 

The boss 

Co-workers 

^Hard to get there 

^Other (explain 



13. Are you satisfied with your job? Yes ^No 

14. Are you planning to change jobs in the near future? ^Yes ^No 

15. How many jobs have you had since living MRC? 

None ^1 2 to 4 5 or more 

16. Since you left MRC vjhat is the longest and shortest time you stayed on your jobs? 

Longest: ^months Shortest: months 



17. 



Please add up the amount of time you v;orked on all the jobs you held since you 
left MRC and write that total time here. 

months 



18. Have you moved since leaving MRC? 



No 



_Yes How many times? 

_Once 4 times 

JTwice 5 times 

3 times More than 5 



19. Has your marital status changed since leaving MRC? 

jGot married 

Got divorced 



_No ^Yes How? 

j3ot separated 
Wife or husband died 



20. Have you been laid up with illness, injury or for any other reason for more than 

two weeks since you left JARC? ^No Yes For how long? 

2 to 4 weeks 4 to 6 weeks 6 weeks to 2 months 

2 to 3 months 3 to 4 months 4 to 6 months 

6 to 9 months 9 to 12 months More than 1 year 



21. What was the reason? 



22. Since you left MRC, have you: 



bought a car 
lost your car 



23. 



Since you left MRC, have you: 



bought a house 
lost your house 




sold your car 
no change 

sold your house 
no change 



MRC Follow"Up Questionraire 



Page 3 



24. 



Since you left MRC, have things: 



gotten better 
gotten worse 
stayed the same 



25. If you checked ’’gotten better” or "gotten worse", please explain how. 



26. Who referred you to MRC? (l) Division of Vocational Rehabilitation (DVR) 

(2) State Services for the Blind (3) Employment Service 

(4) County VJelfare (5) Don't know 

(6) Other (Explain) 

27. Why did you originally come to MRC? 



Answer the following statements by marking an X on the line under True if the state- 
ment is true for you or by marking an X on the line under False if the statement is 
false for you. Mark only one line for each item. 



28. MRC helped me with; 

29. 

30. 

31. 

32. 

33. 

34. 

35. 

36. 

37. 



True 

Getting ajob 

Getting a better job 

Keeping a job 

Training 

Learning more about myself 

Feeling better about myself 

My family 

Friends «... 

Agencies (like DVR, County, etc.) 

Other 

(if True for "other", please explain 



False 



38a MRC did not help me enough 

(if True, what should MC have dons to help you more? ) 



39. Another agency helped me more than MRC did • 

(if True, which agency?) 



40. I think MRC can help mosv. of the people that go there 

While I was at MRC, it was sometimes hard to get along with; 

41. 1 or 2 of the MRC staff members 

42. Almost all of the MRC staff members 



er|c 
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True False 

43. 1 or 2 of the other clients 

44. Almost all of the other clients 

45. 1 or 2 members of my family 

46. Almost my whole family • 

47. What, if anything, did you like about — 



48. What, if anything, did you dislike about fARC? 



49. Have you any other comments? 



Thank you for your cooperation in completing this questionnaire. 



